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SPACE  ACQUISITIONS 

DOD  Needs  to  Take  More  Action  to 
Address  Unrealistic  Initial  Cost  Estimates 
of  Space  Systems 


Why  GAO  Did  This  Study  What  GAO  Found 


Estimated  costs  for  the  Department 
of  Defense’s  (DOD)  major  space 
acquisition  programs  have 
increased  by  about  $12.2  billion 
from  initial  estimates  for  fiscal 
years  2006  through  2011.  Cost 
growth  for  ongoing  Air  Force 
programs  above  initial  estimates 
accounts  for  a  substantial  portion 
of  this  44  percent  increase.  In  light 
of  the  role  that  optimistic 
estimating  is  believed  to  have 
played  in  exacerbating  space 
acquisition  cost  growth,  you 
requested  that  we  examine 
(1)  in  what  areas  space  system 
acquisition  cost  estimates  have 
been  unrealistic  and  (2)  what 
incentives  and  pressures  have 
contributed  to  the  quality  and 
usefulness  of  cost  estimates  for 
space  system  acquisitions. 


What  GAO  Recommends 


GAO  recommends  that  DOD  take  a 
number  of  actions  to  increase  the 
likelihood  that  independent,  more 
realistic  cost  estimates  will  be 
developed  and  utilized. 

DOD  concurred  with  the  overall 
findings  of  this  report  and  provided 
information  on  the  specific  actions 
it  was  already  taking  to  improve 
the  Air  Force’s  cost-estimating 
capability. 


www.gao.gov/cgi-bin/getrpt7GAO-07-96. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  the  link  above. 
For  more  information,  contact  Cristina  T. 
Chaplain  at  (202)  512-4841  or 
chaplainc@gao.gov. 


Costs  for  DOD  space  acquisitions  over  the  past  several  decades  have 
consistently  been  underestimated — sometimes  by  billions  of  dollars.  For 
example,  Space  Based  Infrared  System  High  program  costs  were  originally 
estimated  at  $4  billion,  but  the  program  is  now  estimated  to  cost  over  $10 
billion.  Estimated  costs  for  the  National  Polar-orbiting  Operational  Satellite 
System  program  have  grown  from  almost  $6  billion  at  program  start  to  over 
$11  billion. 

For  the  most  part,  cost  growth  has  not  been  caused  by  poor  cost  estimating, 
but  rather  the  tendency  to  start  programs  before  knowing  whether 
requirements  can  be  achieved  within  available  resources — largely  because  of 
pressures  to  secure  funding.  At  the  same  time,  however,  unrealistic  program 
office  cost  estimates  have  exacerbated  space  acquisition  problems. 
Specifically,  with  budgets  originally  set  at  unrealistic  amounts,  DOD  has  had 
to  resort  to  continually  shifting  funds  to  and  from  programs,  and  such  shifts 
have  had  costly,  reverberating  effects. 

Our  analyses  of  six  ongoing  space  programs  found  that  original  cost 
estimates  were  particularly  unrealistic  about  the  promise  of  savings  from 
increased  contractor  program  management  responsibilities,  the  constancy 
and  availability  of  the  industrial  base,  savings  that  could  be  accrued  from 
heritage  systems,  the  amount  of  weight  growth  that  would  occur  during  a 
program,  the  availability  of  mature  technology,  the  stability  of  funding,  the 
stability  of  requirements,  and  the  achievability  of  planned  schedules.  At 
times,  estimates  that  were  more  realistic  in  these  areas  were  available  to  the 
Air  Force,  but  they  were  not  used. 

Cost-estimating  and  program  officials  we  spoke  with  identified  a  number  of 
factors  that  have  contributed  to  this  condition,  in  addition  to  larger 
pressures  to  produce  low  estimates  that  are  more  likely  to  win  support  for 
funding. 

•  Although  the  National  Security  Space  Acquisition  policy  requires  that 
independent  cost  estimates  be  prepared  by  bodies  outside  the 
acquisition  chain  of  command,  it  does  not  require  that  they  be  relied 
upon  to  develop  program  budgets. 

•  While  the  policy  requires  that  cost  estimates  be  updated  at  major 
acquisition  milestones,  significant  events,  such  as  changes  in  the 
industrial  base  or  funding,  have  occurred  between  milestones. 

•  Within  space  system  acquisitions,  cost-estimating  officials  believe  that 
their  roles  and  responsibilities  are  not  clear  and  the  cost-estimating 
function  is  fragmented. 

•  Cost-estimating  resources  have  atrophied  over  the  years  because  of 
previous  downsizing  of  the  workforce,  making  resources  such  as  staff 
and  data  inadequate  and  the  Air  Force  more  dependent  on  support 
contractors  for  the  estimating  function. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


November  17,  2006 

The  Honorable  Terry  Everett 
Chairman 

The  Honorable  Silvestre  Reyes 
Ranking  Minority  Member 
Subcommittee  on  Strategic  Forces 
Committee  on  Armed  Services 
House  of  Representatives 

Estimated  costs  for  the  Department  of  Defense’s  (DOD)  major  space 
acquisition  programs  have  increased  a  total  of  about  $12.2  billion — or 
nearly  44-percent — above  initial  estimates  for  fiscal  years  2006  through 
2011.  In  some  cases,  current  estimates  of  costs  are  more  than  double  the 
original  estimates.  For  example,  the  Space  Based  Infrared  System  (SBIRS) 
High  program  was  originally  estimated  to  cost  about  $4  billion,  but  is  now 
estimated  to  cost  over  $10  billion.  The  National  Polar-orbiting  Operational 
Satellite  System  (NPOESS)  program  was  originally  estimated  to  cost 
almost  $6  billion  but  is  now  over  $11  billion.  Such  growth  has  had  a 
dramatic  impact  on  DOD’s  overall  space  portfolio.  To  cover  the  added 
costs  of  poorly  performing  programs,  DOD  has  shifted  scarce  resources 
away  from  other  programs,  creating  a  cascade  of  cost  and  schedule 
inefficiencies. 

Our  work  has  identified  a  variety  of  reasons  for  this  cost  growth,  most 
notably  that  weapons  programs  are  incentivized  to  produce  and  use 
optimistic  cost  and  schedule  estimates  in  order  to  successfully  compete 
for  funding  and  that  DOD  starts  its  space  programs  too  early,  that  is, 
before  it  has  assurance  that  the  capabilities  it  is  pursuing  can  be  achieved 
within  available  resources  and  time  constraints.  At  the  same  time, 
however,  this  cost  growth  was  partly  due  to  the  fact  that  DOD  used  low 
cost  estimates  to  establish  programs’  budgets  and  later  found  it  was 
necessary  to  make  funding  shifts  that  had  costly,  reverberating  effects.  In 
2003,  a  DOD  study  of  space  acquisition  problems  found  that  the  space 
acquisition  system  is  strongly  biased  to  produce  unrealistically  low  cost 
estimates  throughout  the  process.  The  study  found  that  most  programs  at 
the  time  of  contract  initiation  had  a  predictable  cost  growth  of  50  to 
100  percent.  The  study  also  found  that  the  unrealistically  low  projections 
of  program  cost  and  lack  of  provisions  for  management  reserve  seriously 
distorted  management  decisions  and  program  content,  increased  risks  to 
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mission  success,  and  virtually  guaranteed  program  delays.  We  have  found 
that  most  of  these  conditions  exist  in  many  DOD  programs. 

Given  concerns  about  the  role  optimistic  cost  estimating  has  played  in 
exacerbating  space  acquisition  problems,  you  requested  that  we  examine 
(1)  in  what  areas  space  system  acquisitions  cost  estimates  have  been 
unrealistic  and  (2)  what  incentives  and  pressures  have  contributed  to  the 
quality  and  usefulness  of  cost  estimates  for  space  system  acquisitions. 

In  conducting  our  work,  we  developed  case  studies  of  six  ongoing  major 
space  acquisition  programs  that  included  analysis  of  cost  and  other 
program  documentation.  These  include  the  Advanced  Extremely  High 
Frequency  (AEHF)  satellite  program  (communications  satellites),  the 
Evolved  Expendable  Launch  Vehicle  (EELV)  (satellite  launch  systems), 
the  Global  Positioning  System  (GPS)  IIF  (navigational  satellites),  the 
National  Polar-orbiting  Operational  Environmental  Satellite  System 
(weather  and  environmental  monitoring  satellites),  the  Space  Based 
Infrared  System  High  (missile  detection  satellites),  and  the  Wideband 
Gapfiller  Satellites  (WGS)  (communication  satellites).  We  also  spoke  with 
officials  from  DOD,  the  Air  Force,  and  contractor  offices  and  analyzed 
DOD  and  Air  Force  acquisition  and  cost-estimating  policies.  In  addition, 
we  obtained  input  on  our  findings  from  a  panel  of  cost-estimating  experts 
who  work  within  the  Office  of  the  Secretary  of  Defense  as  well  as  the  Air 
Force.  Additional  information  on  our  scope  and  methodology  is  in 
appendix  I.  We  conducted  our  work  from  August  2005  to  October  2006  in 
accordance  with  generally  accepted  government  auditing  standards. 


Results  in  Brief 


Our  analyses  of  six  ongoing  space  programs  found  that  original  cost 
estimates  were  unrealistic  in  a  number  of  areas,  specifically,  savings  from 
increased  contractor  program  management  responsibilities,  the  constancy 
and  availability  of  the  industrial  base,  savings  that  could  be  accrued  from 
heritage  systems,  the  amount  of  weight  growth  that  would  occur  during  a 
program,  the  availability  of  mature  technology,  the  stability  of  funding,  the 
stability  of  requirements,  and  the  achievability  of  planned  schedules.  At 
times,  estimates  that  were  more  realistic  in  these  areas  were  available  to 
the  Air  Force,  but  they  were  not  used  so  that  programs  could  sustain 
support  amid  competition  for  funding. 

Cost-estimating  and  program  officials  we  spoke  with  identified  a  number 
of  factors  that  have  contributed  to  low  estimates  in  addition  to  the  larger 
pressures  to  win  support  for  funding.  For  example,  although  the  National 
Security  Space  Acquisition  policy  requires  independent  cost  estimates  that 
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are  prepared  by  bodies  outside  the  acquisition  chain  of  command,  such 
estimates  have  not  always  been  relied  upon  for  program  decisions  or  to 
develop  program  budgets.  In  addition,  while  the  policy  requires  that 
independent  cost  estimates  be  prepared  or  updated  at  major  acquisition 
milestones,  significant  events,  such  as  changes  in  the  industrial  base  or 
funding,  have  occurred  between  milestones.  Moreover,  within  space 
system  acquisitions,  cost-estimating  officials  believe  that  their  roles  and 
responsibilities  are  not  clear,  and  the  cost-estimating  function  is 
fragmented.  Finally,  according  to  Air  Force  officials,  cost-estimating 
resources  have  atrophied  over  the  years  because  of  the  previous 
downsizing  of  the  workforce,  making  resources  such  as  staff  and  data 
inadequate  and  the  Air  Force  more  dependent  on  support  contractors  for 
the  estimating  function. 

While  the  Air  Force  has  taken  steps  recently  to  emphasize  the  use  of 
independent  cost  estimates,  it  has  not  made  additional  changes  needed  to 
enhance  the  quality  of  cost  estimates.  We  are  making  recommendations 
aimed  at  instituting  these  actions.  DOD  agreed  with  most  of  our 
recommendations,  and  is  taking  a  number  of  actions  to  improve  the  Air 
Force’s  cost-estimating  capability  for  space  programs.  DOD  expressed 
concern  that  requiring  officials  involved  in  milestone  decisions  to 
document  and  justify  their  choice  of  cost  estimates  would  reduce  the 
milestone  decision  authority’s  future  decision-making  flexibility.  While  we 
recognize  the  importance  of  decision-making  flexibility,  we  believe  that 
more  transparency  in  DOD’s  decision  making  is  needed  given  the  poor 
foundation  of  choices  made  in  the  past  on  space  programs. 


Background 


Estimates  of  the  total  cost  of  a  program  are  critical  components  in  the 
acquisition  process  because  they  help  decision  makers  decide  among 
competing  options  and  evaluate  resource  requirements  at  key  decision 
points.  All  military  services  prepare  life-cycle  cost  estimates  in  support  of 
their  acquisition  programs  that  attempt  to  identify  all  costs  of  an 
acquisition  program,  from  initiation  through  development,  production,  and 
disposal  of  the  resulting  system  at  the  end  of  its  useful  life.  These 
estimates  serve  two  primary  purposes.  First,  they  are  used  at  acquisition 
program  milestone  and  decision  reviews  to  assess  whether  the  acquisition 
is  affordable  or  consistent  with  the  military  services’  and  DOD’s  overall 
long-range  funding,  investment,  and  force  structure  plans.  Second,  they 
form  the  basis  for  budget  requests  to  Congress.  A  realistic  estimate  of 
projected  costs  makes  for  effective  resource  allocation,  and  it  increases 
the  probability  of  a  project’s  success. 
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The  requirements  and  guidance  for  cost  estimating  are  specified  in  statute 
and  in  DOD  policies.  By  law,  there  is  a  requirement  that  an  independent 
life-cycle  cost  estimate  be  considered  by  the  milestone  decision  authority 
before  approving  system  development  and  demonstration,  or  production 
and  deployment,  of  a  major  defense  acquisition  program.1  The  statute 
requires  DOD  to  prescribe  regulations  governing  the  content  and 
submission  of  such  estimates  and  that  the  estimate  be  prepared  by  (1)  an 
office  or  other  entity  that  is  not  under  the  supervision,  direction,  or 
control  of  the  military  department,  DOD  agency,  or  other  DOD  component 
directly  responsible  for  carrying  out  the  development  or  acquisition  of  the 
program,  or  (2)  by  an  office  or  other  entity  that  is  not  directly  responsible 
for  carrying  out  the  development  or  acquisition  of  the  program  if  the 
decision  authority  for  the  program  has  been  delegated  to  an  official  of  a 
military  department,  DOD  agency,  or  other  DOD  component.2  The  statute 
specifies  that  the  independent  estimate  is  to  include  all  costs  of 
development,  procurement,  military  construction,  and  operations  and 
support,  without  regard  to  funding  source  or  management  control.3  DOD 
policy  assigns  specific  responsibility  for  fulfilling  the  requirement  of  an 
independent  cost  estimate  to  the  Office  of  the  Secretary  of  Defense  Cost 
Analysis  Improvement  Group  (CAIG)  for  any  major  defense  acquisition 
program  and  major  system  that  are  subject  to  review  by  the  Defense 
Acquisition  Board  of  the  Defense  Space  Acquisition  Board.4  These  board 
reviews  address  major  defense  acquisition  programs  (including  space 
programs)  that  are  designated  as  acquisition  category  (ACAT)  ID,  pre¬ 
major  defense  acquisition  programs,  or  ACAT  IC  programs  (see  app.  II  for 
a  description  of  acquisition  categories  ID  and  IC).  The  CAIG  independent 
cost  estimate  is  prepared  for  milestone  (known  as  key  decision  point  in 
space  programs)  B  (program  start,  or  preliminary  design  for  space 
programs),  and  C  (low-rate  initial  production  or  build  approval  for  space 
programs).  In  addition,  the  milestone  decision  authority  may  request  the 
CAIG  to  prepare  other  independent  cost  estimates,  or  conduct  other  ad 
hoc  cost  assessments  for  programs  subject  to  its  review  and  oversight.  The 
CAIG  serves  as  the  principal  advisory  body  to  the  milestone  decision 
authority  on  all  matters  concerning  an  acquisition  program’s  life-cycle 


1 10  U.S.C.  §  2434  (2000). 

2 10  U.S.C.  §  2434(b)(1)(A). 

3  10  U.S.C.  §  2434(b)(1)(B). 

4  DOD  Directive  5000.04,  Cost  Analysis  Improvement  Group  at  f  2  (Aug.  2006);  DOD 
Instruction  5000.2,  Enclosure  6,  Resource  Estimation  (May  2003). 
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cost,  and  is  given  general  responsibilities  for  establishing  DOD  policy 
guidance  on  a  number  of  matters  relating  to  cost  estimating. 

Since  2003,  cost  estimating  for  major  space  system  acquisitions  has  been 
governed  by  the  National  Security  Space  Acquisition  Policy.5 6  Under  this 
policy,  the  CAIG  is  responsible  for  and  leads  the  development  of 
independent  cost  analyses  of  major  space  acquisition  programs.6  Fulfilling 
the  requirement  that  an  independent  cost  estimate  be  developed  by  an 
organization  independent  of  the  program  office  and  the  acquisition  chain 
of  command,  the  CAIG  does  so  in  support  of  a  distinct  Defense  Space 
Acquisition  Board,  with  the  Under  Secretary  of  the  Air  Force  as  the 
milestone  decision  authority.7  The  CAIG  is  to  prepare  independent  cost 
analyses  for  space  acquisition  programs  by  augmenting  its  own  staff  with 
an  independent  team  of  qualified  personnel  from  across  the  space 
community,  including  the  Air  Force  Cost  Analysis  Agency  (AFCAA)  and 
the  cost  estimating  organizations  of  the  Air  Force  Space  Command  and  the 
Air  Force  Space  and  Missile  Systems  Center.  In  addition  to  the 
independent  cost  estimates,  individual  program  offices  also  prepare  cost 
estimates  for  their  acquisition  programs.  The  independent  CAIG  cost 
estimate  is  designed  to  assess  the  program  office  estimate  and  ensure 
realistic  cost  estimates  are  considered.  In  addition,  although  not  required 
in  the  space  acquisition  policy,  in  some  cases  a  cost  analysis  is  prepared 
by  an  Air  Force  service  organization,  such  as  the  Air  Force  Cost  Analysis 
Agency. 


Past  GAO  Findings  on  For  fiscal  years  2006  through  2011,  estimated  costs  for  DOD’s  major  space 

Space  Cost  Growth  acquisition  programs  have  increased  a  total  of  about  $12.2  billion  above 

initial  estimates.  For  example,  the  cost  estimate  for  the  SBIRS  High 
program  rose  from  about  $4  billion  at  the  start  of  development  in  October 
1996  to  over  $10  billion  in  September  2006,  and  costs  are  expected  to  rise 
further.  In  addition,  the  cost  estimate  for  the  NPOESS  program  grew  from 


5  National  Security  Space  Acquisition  Policy  03-01  (revised  December  2004). 

6  National  Security  Space  Acquisition  Policy  at  Appendix  3.2. 

7  Recently,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
withdrew  its  delegation  of  milestone  decision  authority  from  the  Air  Force.  As  a  result, 
although  some  acquisition  authority  was  returned  to  the  Air  Force,  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  is  the  current  milestone  decision 
authority  for  major  space  system  acquisitions.  It  is  not  known  when  or  if  this  role  will  be 
placed  back  within  the  Air  Force. 
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about  $5.9  billion  at  program  start  in  2002  to  nearly  $11.4  billion  currently, 
according  to  the  CAIG’s  latest  estimate. 

Our  past  work  has  identified  a  number  of  causes  behind  the  cost  growth 
and  related  problems,  but  several  consistently  stand  out.  First,  on  a  broad 
scale,  DOD  starts  more  weapon  programs  than  it  can  afford,  creating  a 
competition  for  funding  that  encourages  low  cost  estimating,  optimistic 
scheduling,  overpromising,  suppressing  of  bad  news,  and,  for  space 
programs,  forsaking  the  opportunity  to  identify  and  assess  potentially 
better  alternatives.  Programs  focus  on  advocacy  at  the  expense  of  realism 
and  sound  management.  Invariably,  with  too  many  programs  in  its 
portfolio,  DOD  is  forced  to  continually  shift  funds  to  and  from  programs — 
particularly  as  programs  experience  problems  that  require  more  time  and 
money  to  address.  Such  shifts,  in  turn,  have  had  costly,  reverberating 
effects. 

Second,  as  we  have  previously  testified  and  reported,  DOD  starts  its  space 
programs  too  early,  that  is,  before  it  has  the  assurance  that  the  capabilities 
it  is  pursuing  can  be  achieved  within  available  resources  and  time 
constraints.  This  tendency  is  caused  largely  by  the  funding  process,  since 
acquisition  programs  attract  more  dollars  than  efforts  concentrating  solely 
on  proving  technologies.  Nevertheless,  when  DOD  chooses  to  extend 
technology  invention  into  acquisition,  programs  experience  technical 
problems  that  require  large  amounts  of  time  and  money  to  fix.  Moreover, 
when  this  approach  is  followed,  cost  estimators  are  not  well  positioned  to 
develop  accurate  cost  estimates  because  there  are  too  many  unknowns. 
Put  more  simply,  there  is  no  way  to  estimate  how  long  it  would  take  to 
design,  develop,  and  build  a  satellite  system  when  critical  technologies 
planned  for  that  system  are  still  in  relatively  early  stages  of  discovery  and 
invention. 

A  companion  problem  for  space  systems  is  that  programs  have  historically 
attempted  to  satisfy  all  requirements  in  a  single  step,  regardless  of  the 
design  challenge  or  the  maturity  of  the  technologies  necessary  to  achieve 
the  full  capability.  Increasingly,  DOD  has  preferred  to  make  fewer,  but 
heavier,  large  and  complex  satellites  that  perform  a  multitude  of  missions 
rather  than  larger  constellations  of  smaller,  less  complex  satellites  that 
gradually  increase  in  sophistication.  This  has  stretched  technology 
challenges  beyond  current  capabilities  in  some  cases  and  vastly  increased 
the  complexities  related  to  software — a  problem  that  affected  SBIRS  High 
and  AEHF,  for  example. 
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In  addition,  several  of  the  space  programs  included  in  our  case  studies, 
began  in  the  late  1990s,  when  DOD  structured  contracts  in  a  way  that 
reduced  oversight  and  shifted  key  decision-making  responsibility  onto 
contractors.  This  approach — known  as  Total  System  Performance 
Responsibility,  or  TSPR — was  intended  to  facilitate  acquisition  reform  and 
enable  DOD  to  streamline  a  cumbersome  acquisition  process  and  leverage 
innovation  and  management  expertise  from  the  private  sector.  However, 
DOD  later  found  that  this  approach  magnified  problems  related  to 
requirements  creep  and  poor  contractor  performance.  In  addition,  under 
TSPR,  the  government  decided  not  to  obtain  certain  cost  data,  a  decision 
that  resulted  in  the  government  having  even  less  oversight  of  the  programs 
and  limited  information  from  which  to  manage  the  programs.  Further,  the 
reduction  in  government  oversight  and  involvement  led  to  major 
reductions  in  various  government  capabilities,  including  cost-estimating 
and  systems-engineering  staff.  The  loss  of  cost-estimating  and  systems- 
engineering  staff  in  turn  led  to  a  lack  of  technical  data  needed  to  develop 
sound  cost  estimates. 

Our  reviews  have  identified  additional  factors  that  have  contributed  to 
space  cost  growth,  though  less  directly.  These  include  consolidations 
within  the  defense  supplier  base  for  space  programs,  the  diverse  array  of 
officials  and  organizations  involved  with  space  programs,  short  tenures  for 
top  leadership  and  program  managers,  as  well  as  capacity  shortfalls  that 
have  constrained  DOD’s  ability  to  optimize  and  oversee  its  space 
programs.  A  section  at  the  end  of  this  report  lists  prior  relevant  GAO 
reports. 


Program  Office  Cost 
Estimates  on  Space 
Programs  Not 
Realistic 


Our  case  study  analyses  found  that  program  office  cost  estimates — and 
more  specifically,  the  assumptions  upon  which  those  estimates  were 
based — have  been  unrealistic  in  eight  areas,  many  of  which  are 
interrelated.  In  some  cases,  such  as  assumptions  regarding  weight  growth 
and  the  ability  to  gain  leverage  from  heritage,  or  legacy,  systems,  past 
experiences  or  contrary  data  were  ignored.  In  other  cases,  such  as  when 
contractors  were  given  more  program  management  responsibility,  as  with 
TSPR,  or  when  growth  in  the  commercial  market  was  predicted, 
estimators  assumed  that  promises  of  reduced  cost  and  schedule  would  be 
borne  out  and  did  not  have  the  benefit  of  experience  to  factor  into  their 
work.  We  also  identified  flawed  assumptions  that  reflected  deeper  flaws  in 
acquisition  strategies  or  development  approaches.  For  example,  five  of  six 
programs  we  reviewed  assumed  technology  would  be  sufficiently  mature 
when  needed,  even  though  the  programs  began  without  a  complete 
understanding  of  how  long  it  would  take  or  how  much  it  would  cost  to 
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ensure  technologies  could  work  as  intended.  In  four  programs,  estimators 
assumed  there  would  be  few  delays,  even  though  programs  were  adopting 
highly  aggressive  schedules  while  simultaneously  attempting  to  make 
ambitious  leaps  in  capability.  In  four  programs,  estimators  assumed 
funding  would  stay  constant,  even  though  space  and  weapon  programs 
frequently  experience  funding  shifts  and  the  Air  Force  was  in  the  midst  of 
starting  a  number  of  costly  new  space  programs  to  replenish  older 
constellations. 

Table  1  highlights  major  areas  where  program  officials  were  too  optimistic 
in  their  assumptions  for  the  six  space  system  acquisitions  we  examined  or 
where  additional  evidence  showed  the  estimate  was  unrealistic.  In  some 
cases,  programs  may  have  experienced  problems  related  to  one  of  the 
categories,  but  we  did  not  have  evidence  to  show  the  original  assumptions 
were  optimistic. 


Table  1 :  Areas  Where  Program  Officials  Were  Too  Optimistic  in  Their  Assumptions 


Space  programs  affected 

Optimistic  assumptions 

AEHF 

EELV 

GPS  IIF 

NPOESS  SBIRS  High 

WGS 

Industrial  base  would  remain  constant  and  available 

X 

X 

X 

X 

X 

Technology  would  be  mature  enough  when  needed 

X 

X 

X 

X 

X 

TSPR  would  reduce  costs  and  schedule 

X 

X 

X 

X 

Savings  would  occur  from  experience  on  heritage  systems 

X 

X 

X 

X 

No  weight  growth  would  occur 

X 

X 

X 

X 

Funding  stream  would  be  stable 

X 

X 

X 

X 

An  aggressive  schedule 

X 

X 

X 

X 

No  growth  in  requirements 

X 

X 

X 

Source:  This  table  is  based  on  conversations  with  program  and  contracting  officials  and  analysis  of  data  they  provided.  In  some  cases, 
we  made  our  own  designations  based  on  our  prior  findings. 


•  Assumptions  about  the  space  industrial  base:  Five  programs 

experienced  challenges  due  to  assumptions  that  were  made  about  the 
availability  and  constancy  of  the  industrial  base.  When  cost  estimates 
for  some  of  these  programs  were  developed,  cost  estimators  assumed 
the  programs  would  gain  leverage  from  the  commercial  satellite 
market,  which,  at  the  time  the  programs  were  initiated,  was  widely 
expected  to  continue  to  grow.  In  the  EELV  program,  for  instance,  the 
original  contracting  concept  was  for  the  Air  Force  to  piggyback  on  the 
anticipated  launch  demand  of  the  commercial  sector.  Furthermore,  the 
Air  Force  assumed  that  it  would  benefit  financially  from  competition 
among  commercial  vendors.  However,  the  commercial  demand  never 
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materialized,  and  the  government  was  forced  to  bear  the  cost  burden  of 
maintaining  the  industrial  base  in  order  to  maintain  launch  capability, 
and  assumed  savings  from  competition  were  never  realized.  In  other 
cases,  programs  experienced  unanticipated  problems  resulting  from 
consolidations  in  the  supplier  base.  For  example,  contractors  took 
cost-cutting  measures  that  reduced  the  quality  of  parts.  Contractors 
also  lost  key  technical  personnel  as  they  consolidated  development  and 
manufacturing  facilities. 

•  Assumptions  about  technology  maturity:  In  five  of  the  six  space 
system  acquisition  programs,  when  cost  estimates  were  developed, 
program  officials  and  cost  estimators  assumed  that  technologies 
critical  to  the  programs  would  be  mature  and  available — even  though 
the  programs  began  without  a  complete  understanding  of  how  long  or 
how  much  it  would  cost  to  ensure  technologies  could  work  as 
intended.  Invariably,  after  the  programs  began  and  as  their 
development  continued,  the  technology  issues  ended  up  being  more 
complex  than  initially  believed.  For  example,  on  the  NPOESS  program, 
DOD  and  the  Department  of  Commerce  committed  funds  for  the 
development  and  production  of  satellites  before  the  technology  was 
mature — only  1  of  14  critical  technologies  was  mature  at  program 
initiation  and  1  technology  was  determined  to  be  less  mature  after  the 
contractor  conducted  more  verification  testing.  The  program  has  since 
been  beset  by  significant  cost  increases  and  schedule  delays  due  in  part 
to  technical  problems,  such  as  the  development  of  key  sensors.  On  the 
GPS  IIF  program,  the  cost  estimate  was  built  on  the  assumption  that 
the  military  code  signal  being  developed  would  fit  on  a  single 
microchip.  However,  once  development  started,  interface  issues  arose 
and  the  subcontractor  had  to  move  to  a  two-microchip  design,  which 
took  8  months  to  resolve  and  increased  cost  to  the  program. 

•  Assumptions  about  TSPR  savings:  Four  programs  we  examined 
assumed  that  there  would  be  significant  savings  associated  with 
adopting  the  TSPR  policy.  For  example,  while  TSPR  was  supposed  to 
relieve  contractors  of  unnecessary  oversight,  the  government  assumed 
that  the  contractors  would  still  maintain  sufficient  systems  engineering 
and  program  management  levels  by  following  standard  practices  to 
provide  oversight  of  their  subcontractors  and  vendors.  However,  for  a 
variety  of  reasons,  the  savings  never  materialized.  For  instance,  it  was 
believed  that  by  giving  more  program  management  responsibility  to 
contractors  and  increasing  use  of  commercial  equipment,  the 
government  could  reduce  the  number  of  in-house  systems  engineers — 
who  normally  help  the  government  define  its  requirements  by  analyzing 
differences  between  customer  needs  and  technical  possibilities  and 
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analyze  progress  in  development.  Ultimately,  the  reduction  in  systems 
engineering  staff  resulted  in  cost  growth  as  the  programs  experienced 
technical  and  quality  problems  that  the  government  was  no  longer  in  a 
position  to  detect  and  prevent.  Programs  also  came  to  realize  that 
commercial  parts  being  relied  on  were  not  always  suitable  for  their 
efforts,  and  had  to  resort  to  costly  measures  to  address  this  problem.  In 
addition,  in  implementing  TSPR,  the  government  initially  entered  into 
contracts  that  did  not  allow  it  to  obtain  certain  cost  data  from  the 
contractors  (e.g.,  contractor  cost  data  reports  and  contractor 
performance  reports),  even  though  such  data  are  critical  for  cost 
estimators  to  develop  sound  cost  estimates  and  important  for  the 
government  to  maintain  adequate  insight.  This  was  the  case  for  EELV 
and  GPS  IIF — both  of  which  have  either  been  restructured  or  are  now 
planning  to  issue  follow-on  contracts  that  will  require  cost  and  pricing 
data  and  earned  value  management  data.  It  should  be  noted  that  the  Air 
Force  has  since  recognized  problems  related  to  its  implementation  of 
TSPR  and  rejected  it  as  a  recommended  approach. 

•  Assumptions  about  savings  from  heritage  systems:  Four  programs 
assumed  that  they  would  be  able  to  gain  leverage  from  legacy  satellite 
systems  and  save  costs,  but  as  the  programs  continued  and  more 
knowledge  was  gained  about  the  requirements  and  the  technologies 
needed  to  meet  the  requirements,  DOD  discovered  that  the  legacy 
systems  could  not  be  relied  on,  as  initially  believed,  and  the  savings 
were  not  realized.  In  addition,  SBIRS  High  and  WGS,  for  example,  had 
all  planned  to  gain  leverage  from  commercial  satellite  development 
efforts  because  the  government  had  planned  to  use  portions  of  these 
satellites  as  lessons  already  learned  in  order  to  obtain  design  savings. 
However,  when  hardware  and  software  development  advances  were 
slowed  as  a  result  of  the  Internet  sector  economic  downturn,  the 
government  had  to  carry  more  design  and  development  costs  than 
anticipated. 

•  Assumptions  about  weight  growth:  Four  case  study  programs 
assumed  no  weight  growth,  which  is  among  the  highest  drivers  of  cost 
growth  for  space  systems,  would  occur  despite  leaps  hoped  for  in 
technology  and  experiences  in  past  programs.  For  example,  the  SBIRS 
High  program  assumed  little  to  no  weight  growth,  but  the  weight  of  the 
satellite  spacecraft  eventually  grew  by  more  than  59  percent,  while 
payload  aboard  the  spacecraft  grew  by  44  percent.  Moreover,  with  such 
considerable  weight  growth,  the  program  could  no  longer  rely  on  the 
commercial  bus  it  had  originally  selected  for  this  acquisition,  and 
instead  had  to  develop  a  custom  satellite  bus — a  more  expensive 
endeavor. 
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•  Assumptions  about  funding:  Space  programs  frequently  experienced 
funding  shifts.  Moreover,  at  the  time  the  Air  Force  undertook  the 
programs  included  in  our  case  studies,  it  was  attempting  to  replenish 
several  older  satellite  constellations,  which  put  further  stress  on  its 
total  investment  in  space.  Despite  this  condition,  when  making 
estimates  on  four  programs  we  reviewed,  cost  estimators  assumed  that 
program  budgets  would  remain  stable.  As  the  programs  progressed 
through  the  acquisition  cycle,  they  experienced  changes  to  their 
funding  stream,  which  created  program  instability  and  cost  growth  due 
to  the  stopping  and  starting  of  activities.  Cost  estimators  and  program 
officials  we  interviewed  generally  agreed  that  space  programs  are  not 
often  fully  funded  and  that  their  programs  have  experienced  shifts  in 
funding.  However,  they  could  not  separate  the  ultimate  effects  of 
funding  shifts,  since  the  programs  were  concurrently  experiencing 
other  problems,  such  as  technical  or  design  problems,  which  were  also 
adding  costs,  and  these  funding  cuts  led  to  other  decisions  that  had 
reverberating  consequences.  For  example,  in  some  cases,  programs 
abandoned  their  original  plans  to  purchase  satellites  in  one 
procurement  in  favor  of  individual  orders  in  an  effort  to  address  a 
funding  cut.  While  this  decision  enabled  the  programs  to  continue  in 
the  short  term,  it  had  significant  long-term  consequences  on  program 
costs  since  the  price  of  each  satellite  substantially  increased  with  the 
change  to  individual  orders.  In  previous  testimony  and  reports,  we  have 
stressed  that  DOD  could  avoid  the  need  to  make  costly  funding  shifts 
by  developing  an  overall  investment  strategy  that  would  prioritize 
systems  in  its  space  portfolio  with  an  eye  toward  balancing 
investments  between  legacy  systems  and  new  programs  as  well  as 
between  science  and  technology  programs  and  acquisition  investments. 
Such  prioritizing  would  also  reduce  incentives  to  produce  low 
estimates. 

•  Assumptions  about  schedules:  Four  case  study  programs  assumed  that 
compressed  schedules  being  proposed  could  be  achieved — even 
though  the  programs  were  pursuing  ambitious  leaps  in  capability  or 
attempting  new  approaches,  such  as  using  commercial  equipment  for 
military  purposes.  Moreover,  in  some  cases,  DOD  had  data  available 
demonstrating  such  schedules  were  not  realistic.  In  one  case  study 
program,  WGS,  the  request  for  proposals  specified  that  the  budget 
available  was  $750  million  for  three  satellites  plus  ground  control  with 
a  schedule  constraint  of  36  months.  On  the  basis  of  these  requirements, 
competing  contractors  were  asked  to  offer  maximum  capacity, 
coverage,  and  connectivity  via  a  contract  that  would  make  use  of 
existing  commercial  practices  and  technologies.  This  aggressive 
schedule  was  never  achieved.  Instead,  problems  due  to  higher  design 
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complexity  and  supplier  quality  issues  have  caused  the  WGS  schedule 
to  stretch  to  78  months  for  the  first  expected  launch.  Historically,  the 
Air  Force  has  required  between  55  and  79  months  to  build  satellites 
similar  to  WGS,  so  while  the  schedule  slip  is  within  the  expected  range, 
the  original  36-month  schedule  was  optimistic  and  not  based  on 
realistic  data.  For  AEHF,  the  program  accelerated  its  schedule  in 
response  to  a  potential  gap  in  satellite  coverage  due  to  the  launch 
failure  of  the  third  Milstar  satellite.  However,  when  the  funding  needed 
to  achieve  the  acceleration  was  not  delivered,  the  program  experienced 
cost  and  schedule  delays.  Again,  because  these  assumptions  were 
made  before  enough  information  about  the  development  was  available, 
the  assumptions  did  not  hold  up,  and  the  programs  experienced  cost 
and  schedule  growth  as  a  result. 

•  Assumptions  about  requirements  growth:  Three  programs — AEHF, 
GPS  IIF,  and  SBIRS  High — did  not  assume  any  requirements  growth, 
even  though  there  was  a  risk  of  growth  because  of  the  variety  of 
stakeholders  involved.  High-level  requirements  for  the  SBIRS  High 
program — which  is  being  developed  to  improve  missile  warning, 
missile  defense,  technical  intelligence,  and  battle  space 
characterization — have  remained  stable  since  the  program  began,  but 
prior  DOD  studies  have  found  that  lower-level  requirements  were  in 
flux  and  mismanaged  until  the  program  was  restructured  in  1999. 
According  to  DOD  studies,  this  was  partially  due  to  the  TSPR 
approach,  which  placed  too  much  responsibility  on  contractors  to 
negotiate  these  requirements;  the  broad  customer  base  for  SBIRS;  and 
the  ambitious  nature  of  the  program  to  begin  with.  To  illustrate,  the 
SBIRS  High  program  has  19  key  performance  parameters  to  satisfy — 
nearly  five  times  more  than  the  typical  DOD  space  program.  In 
addition,  there  are  over  12,600  requirements  that  the  program  must 
address,  and  to  date,  requirements  for  external  users  have  not  been 
fully  defined.  DOD  has  since  realized  that  responsibility  for  setting 
lower-level  requirements  should  rest  with  the  government  and  has 
taken  actions  to  add  more  discipline  to  the  requirements-setting 
process.  In  another  example,  GPS  IIF  was  intended  to  follow  on  to  the 
GPS  II  program,  yet  shortly  after  the  contract  was  awarded,  the 
government  added  the  requirement  for  an  additional  auxiliary  payload. 
This  requirement  caused  the  satellite  design  to  be  larger  than  originally 
planned,  and  this,  in  turn,  required  a  larger  launch  vehicle. 
Requirements  for  more  robust  jamming  capability  to  secure  satellite 
transmissions  were  also  added.  Changes  from  a  two-panel  to  a  three- 
panel  solar  array  design  and  flexible  power  were  necessary  to  allow  for 
more  power  and  thermal  capability  requirements. 
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Appendix  III  contains  additional  detailed  examples  of  instances  where 
program  officials  were  too  optimistic  in  their  assumptions  for  the  six 
space  system  acquisitions  we  examined. 


Various  Incentives 
and  Pressures  within 
DOD  Have 
Contributed  to  Cost- 
Estimating 
Weaknesses 


Various  incentives  and  pressures  within  DOD  have  contributed  to 
optimistic  program  office  cost  estimates  for  space  system  acquisitions.  As 
noted  earlier,  our  prior  work  has  found  that  programs  are  incentivized  to 
produce  optimistic  estimates  in  order  to  gain  approval  for  funding.  At 
present,  DOD  does  not  have  a  long-term  investment  strategy  that  would 
prioritize  its  investments  and,  in  turn,  reduce  pressures  associated  with 
competition  for  funding.  A  2003  DOD  study  on  crosscutting  problems 
affecting  space  acquisitions,  known  as  the  Young  Panel  report,  also  found 
that  the  space  acquisition  system,  in  particular,  is  strongly  biased  to 
produce  unrealistically  low  cost  estimates  throughout  the  process; 
advocacy  tends  to  dominate,  and  a  strong  motivation  exists  to  minimize 
program  cost  estimates,  and  proposals  from  competing  contractors 
typically  reflected  the  minimum  program  content  and  a  price  to  win.  In 
responding  to  the  Young  Panel  report  as  well  as  our  prior  reports,  DOD 
officials  have  not  disputed  the  need  for  long-term  investment  planning  or 
that  programs  are  incentivized  to  produce  low  estimates. 


In  conducting  this  review,  we  asked  cost  estimators,  program  managers, 
industry  officials,  and  higher-level  oversight  officials  what  additional 
impediments  there  were  to  sound  cost  estimating  for  space.  Their 
responses  included  that  (1)  there  is  little  accountability  for  producing 
realistic  program  office  estimates — among  both  program  managers  and 
estimators;  (2)  estimates  produced  within  program  offices  are  more  often 
used  to  set  budgets  than  estimates  produced  by  independent  estimators; 
(3)  even  though  space  programs  experience  frequent  changes, 
independent  cost  estimates  are  not  updated  for  years  at  a  time;  (4)  cost- 
estimator  roles  and  responsibilities  are  not  clear  and  the  cost-estimating 
function  is  fragmented;  and  (5)  there  are  not  enough  in-house  government 
cost  estimators  or  sufficient  data  to  support  their  work. 


Accountability  Is  Lacking  It  is  difficult  for  cost  estimators  to  be  held  accountable  for  the  estimates 

they  develop  because  program  decision  makers  are  rarely  held 
accountable  for  the  estimates  they  use  to  establish  program  budgets.  This, 
coupled  with  the  pressure  to  compete  for  funding,  invites  program 
officials  to  accept  optimistic  assumptions  and  ignore  risk  and  reality  when 
developing  cost  estimates. 
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This  view  was  also  expressed  by  many  DOD  program  managers  we 
interviewed  for  a  2005  review  on  program  management  best  practices.8 
While  many  program  managers  told  us  that  they  personally  held 
themselves  accountable,  many  also  commented  that  it  is  difficult  to  be 
accountable  when  so  much  is  outside  their  control.  During  our  focus 
groups,  program  managers  cited  sporadic  instances  when  program 
managers  were  removed  from  their  positions  or  forced  to  retire  if 
programs  came  in  over  cost  or  schedule,  but  they  also  cited  instances 
when  a  program  manager  was  promoted  even  though  the  program  was 
experiencing  difficulties. 


Independent  Estimates  We  found  examples  from  our  closer  examinations  of  the  AEHF,  NPOESS, 

Not  Always  Relied  Upon  and  SBIRS  High  programs  where  independent  cost  estimates  were  not 

relied  upon  by  program  decision  makers.  Independent  estimates  for  these 
space  system  acquisitions  forecasted  considerably  higher  costs  and 
lengthier  schedules  than  program  office  or  service  cost  estimates.  Yet  the 
milestone  decision  authorities  used  program  office  estimates  or  even 
lower  estimates  instead  of  the  independent  estimates  to  establish  budgets 
for  their  programs.  DOD’s  current  space  acquisition  policy  requires  that 
independent  cost  estimates  be  prepared  by  bodies  outside  the  acquisition 
chain  of  command,  and  be  considered  by  program  and  DOD  decision 
makers.  However,  the  policy  does  not  require  that  the  independent 
estimates  be  relied  upon  to  set  budgets,  only  that  they  be  considered  at 
key  acquisition  decision  points. 

•  AEHF:  In  2004,  AEHF  program  decision  makers  relied  upon  the 
program  office  cost  estimate  rather  than  the  independent  estimate 
developed  by  the  CAIG  to  support  the  production  decision  for  the 
AEHF  program — which  was  more  than  $2  billion  higher.  At  that  time, 
the  AEHF  program  office  estimated  the  system  would  cost  $6  billion. 
This  was  based  on  the  assumption  that  AEHF  would  have  10  times 
more  capacity  than  the  predecessor  satellite — Milstar — but  at  half  the 
cost  and  weight.  The  CAIG  believed  that  this  assumption  was  overly 
optimistic  given  that  the  AEHF  weight  had  more  than  doubled  since  the 
program  began  in  1999  to  obtain  the  desired  increase  in  data  rate.  The 
latest  program  office  estimate  for  AEHF  is  $6.1  billion. 


8  GAO,  Best  Practices:  Better  Support  of  Weapon  System  Program  Manager's  Needed  to 
Improve  Outcomes,  GAO-06-110  (Washington,  D.C.:  Nov.  30,  2005). 
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Table  2:  Comparison  of  2004  AEHF  Program  Office  and  Independent  Cost  Estimates 


Independent  cost  estimate 

Program  office 
estimate 

AFCAA 

CAIG 

Difference 

Latest 

program 

office 

estimate 

$6  billion 

AFCAA  worked 
jointly  with  the 

$8.7  billion 

44% 

$6.1  billion 

CAIG  to  develop 
the  independent 
estimate 


Source:  CAIG  and  GAO  analysis. 

Note:  Estimates  are  in  fiscal  year  2006  dollars. 


•  NPOESS:  In  2003,  to  support  the  NPOESS  development  decision, 

government  decision  makers  relied  on  the  program  office’s  $7.2  billion 
cost  estimate  rather  than  the  $8.8  billion  independent  cost  estimate 
presented  by  the  Air  Force  Cost  Analysis  Agency.  AFCAA  based  its 
estimate  on  an  analysis  of  historical  data  from  satellite  systems, 
independent  software  and  hardware  models,  and  a  risk  simulation 
model  using  input  from  30  independent  engineers.  The  program  office 
relied  largely  on  the  contractor’s  proposal  as  well  as  on  an  unrealistic 
estimate  of  what  it  would  cost  to  integrate  the  payloads  onto  the 
satellite  bus.  The  program  has  encountered  many  problems  as  a  result 
of  these  optimistic  assumptions,  and  costs  have  risen  to 
$11.4  billion,  based  on  the  latest  program  office  cost  estimate. 
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Table  3:  Comparison  of  2003  NPOESS  Program  Office  and  Independent  Cost 
Estimates 


Independent  cost  estimate 

Latest 

program 

office 

Program  office  estimate  AFCAA  CAIG  Difference  estimate 


$7.2  billion  (based  on  $8.8  billion  23%  $11.4  billion 

planned  purchase  of  six  (based  on 

satellites)  planned 

purchase  of 
four 
satellites) 


Source:  CAIG  and  GAO  analysis. 

Note:  Estimates  are  in  fiscal  year  2006  dollars.  The  CAIG  was  not  involved  in  preparing  the  2003 
independent  cost  estimate. 


SBIRS  High.  On  the  SBIRS  High  program,  the  program  office  and  AFCAA 
predicted  cost  growth  as  early  as  1996,  when  the  program  was  initiated. 
While  both  estimates  at  that  time  were  close,  approximately  $6.6  billion, 
both  were  much  higher  than  the  contractor’s  estimated  costs.  The  program 
was  subsequently  estimated  to  cost  $3.6  billion  by  the  program  office, 
almost  $2  billion  less  than  the  original  AFCAA  or  program  office  estimate. 
The  program  office  and  contractor  ultimately  assumed  savings  under 
TSPR  that  did  not  materialize.  For  instance,  with  this  approach,  the  SBIRS 
High  contractor  used  far  fewer  systems  engineers  than  historical  data 
show  have  been  used  for  similar  programs.  To  achieve  savings,  the 
contractor  dropped  important  systems  engineering  tasks  such  as 
verification  and  cycling  of  requirements.  The  lack  of  systems  engineering 
resulted  in  latent  design  flaws  that  required  more  integration  and  testing 
when  components  failed  initial  testing. 


Table  4:  Comparison  of  1996  SBIRS  High  Program  Office  Cost  Estimate  and 
Independent  Cost  Estimate 

Program  office 
estimate 

AFCAA  independent 
cost  estimate 

Total  program 
funding 

Latest  program 
office  estimate 

$5.7  billion  (based 
on  a  planned 
purchase  of  five 
satellites) 

$5.6  billion 

$3.6  billion 

$10.2  billion  (based 
on  a  planned 
purchase  of  three 
satellites) 

Source:  AFCAA  and  Air  Force  documentation  and  GAO  analysis. 


Note:  Estimates  are  in  fiscal  year  2006  dollars. 
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We  were  informed  by  the  CAIG  that  independent  cost  estimates  are  rarely 
used  by  the  services  to  develop  budgets  for  acquisition  programs.  Because 
CAIG  estimates  are  seldom  used  and  the  program  offices  know  this, 
officials  we  spoke  with  believe  that  there  is  no  incentive  on  the  part  of 
program  offices  to  change  their  approach  to  cost  estimating.  According  to 
a  senior  CAIG  official,  program  managers  often  promise  to  meet  the 
maximum  amount  of  requirements  for  the  least  cost.  These  program 
officials  would  rather  rely  on  optimistic  cost  estimates  from  the 
contractors  because  these  estimates  most  likely  align  with  program 
objectives. 

Appendix  IV  contains  detailed  examples  of  where  program  and  cost¬ 
estimating  officials  disagreed  on  estimates. 


Independent  Cost 
Estimates  Not  Updated 
Frequently  Enough  to 
Account  for  Significant 
Events  and  Changes 


It  is  possible  for  space  programs  to  continue  for  years — as  many  as  4 
years — without  updates  of  independent  cost  estimates  and  to  see  changes 
within  that  span  of  time  that  have  had  a  substantial  impact  on  cost — 
including  changes  in  requirements,  changes  in  planned  quantities,  funding 
instability,  design  changes,  quality  variances  resulting  from  rework, 
manufacturing  or  engineering  changes,  changes  in  supply  chain  and 
logistics  management  and  support,  technology-related  problems,  among 
others.  At  times,  the  only  mechanism  that  forced  an  updated  estimate  was 
DOD  policy  that  the  CAIG  support  the  Nunn-McCurdy  certification 
process  for  programs  breaching  a  certain  unit  cost  threshold.9  Under  this 
policy,10  the  CAIG  provides  the  Under  Secretary  with  a  recommendation 
concerning  the  reasonableness  of  the  most  recent  unit  cost  estimates  by 
the  program. 

Because  space  programs  tend  to  experience  such  changes  after  program 
start,  some  officials  we  spoke  with  in  the  DOD  space  cost-estimating 
community  believe  that  independent  cost  estimates  should  be  updated 
more  frequently.  Opinions  differ  as  to  the  frequency  and  phasing  of  these 


10  U.S.C.  §  2433.  This  oversight  mechanism  originated  with  the  Department  of  Defense 
Authorization  Act,  1982.  It  was  made  permanent  in  the  following  year’s  authorization  act 
and  has  been  amended  several  times.  Generally,  the  law  requires  DOD  to  review  programs 
and  report  (and  in  some  cases  submit  a  certification)  to  Congress  whenever  cost  growth 
reaches  specified  thresholds.  The  statute  is  commonly  known  as  Nunn-McCurdy,  based  on 
the  names  of  the  sponsors  of  the  original  legislation. 

10  DOD  Directive  5000.04  at  If  4.8. 
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non  milestone  estimates,  assessments,  or  reviews.  A  CAIG  official 
suggested  updating  cost  estimates  about  every  18  to  24  months,  while 
AFCAA  officials  suggested  annually  to  correspond  with  the  annual 
budgeting  cycle.  The  current  space  acquisition  policy  requires  only  one 
independent  cost  estimate  after  critical  design  review,  but  CAIG  officials 
noted  that  years  can  go  by  between  critical  design  review  and  program 
completion,  during  which  time  programs  have  historically  experienced 
substantial  changes. 

Figure  1  illustrates  significant  changes  that  took  place  on  the  SBIRS  High 
program  both  before  and  after  critical  design  review. 


Figure  1:  Key  Events  and  Funding  Shifts  That  Occurred  between  Estimates  for  SBIRS  High 


K  Independent  cost  estimates 

A  Change  in  program  scope  and  content  such  as  requirments  and  quantity  changes 

•  Programmatic  changes  such  as  funding  reductions 

■  Quality  variances  resulting  from  rework  or  manufacturing  or  engineering  changes 

♦  Unforeseen  problems  such  as  teachnical  risks  and  forgotten  items 


Source:  GAO  analysis  of  SBIRS  High  program  data. 


Cost-Estimating  Roles  and 
Responsibilities  Are 
Unclear 


Air  Force  cost-estimating  officials  believe  that  their  roles  and 
responsibilities  are  not  clear  and  that  the  cost-estimating  function  is  too 
fragmented.  Some  also  asserted  that  the  cost-estimating  function  within 
the  space  community  would  be  stronger  if  estimators  themselves  were 
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centralized  outside  the  acquisition  chain  of  command  so  that  they  would 
not  be  biased  or  pressured  by  program  office  leadership  to  produce 
optimistic  estimates. 

In  an  attempt  to  make  the  most  efficient  use  of  the  limited  cost  estimate 
expertise  for  DOD  space  system  acquisitions,  the  space  acquisition  policy 
called  on  the  CAIG  to  augment  its  own  staff  with  cost-estimating 
personnel  drawn  from  across  the  community  to  serve  as  team  members 
when  it  developed  independent  estimates.  Members  were  to  include  the 
intelligence  community’s  cost  analysis  improvement  group,  the  Air  Force 
Cost  Analysis  Agency,  the  National  Reconnaissance  Office  (NRO)  Cost 
Group,  the  Office  of  the  Deputy  Assistant  Secretary  of  the  Army  for  Cost 
and  Economics,  the  Naval  Center  for  Cost  Analysis,  the  cost-estimating 
organizations  of  the  Air  Force  Space  Command,  Air  Force  Space  and 
Missile  Systems  Center,  and  the  Space  and  Naval  Warfare  Systems 
Command. 

At  this  time,  however,  there  are  still  significant  disconnects  in  views  about 
roles  and  responsibilities.  Officials  who  reside  in  the  acquisition  chain  of 
command — the  Air  Force  Space  and  Missile  Systems  Center — believe  that 
because  the  program  executive  officer  and  the  program  managers  are 
responsible  for  executing  the  programs,  they  are  also  solely  responsible 
for  the  cost  estimates  for  the  program.  On  the  other  hand,  Air  Force  cost 
estimators  outside  the  acquisition  chain  of  command — the  Air  Force  Cost 
Analysis  Agency — believe  they  also  hold  some  responsibility  to  ensure  the 
quality  and  consistency  of  cost  estimates  and  to  produce  independent  cost 
estimates  for  consideration  by  Air  Force  decision  makers.  However, 
according  to  officials  within  the  Space  and  Missile  Systems  Center’s  (SMC) 
cost-estimating  group  and  AFCAA,  the  SMC  cost-estimating  group  sees  no 
role  for  AFCAA  in  developing  program  or  Air  Force  cost  estimates  and  has 
rejected  assistance  from  AFCAA.  According  to  Air  Force  officials,  until  a 
clearer  distinction  of  roles  and  responsibilities  is  defined  by  Air  Force 
leadership,  issues  of  conflicting  policy  interpretation  and  implementation 
will  remain.  It  is  also  possible  that  these  disconnects  have  been 
exacerbated  by  the  perception  that  these  two  communities  are  competing 
for  responsibility. 

In  addition,  according  to  a  senior  CAIG  official,  the  collaborative  process 
for  developing  independent  estimates  has  not  been  achieved  as 
envisioned — principally  because  those  who  should  be  involved  have  not 
seen  their  involvement  as  a  priority,  and  those  who  have  been  involved 
have  required  a  lot  of  extra  training  to  be  able  to  make  valuable 
contributions.  Moreover,  because  the  various  cost-estimating 


Page  19 


GAO-07-96  Space  Acquisitions 


organizations  each  have  different  customers,  agendas,  and  approaches  to 
developing  cost  estimates,  these  differences  have  made  it  difficult  for 
them  to  work  as  a  cohesive  team. 


Cost-Estimating  Resources  Air  Force  space  cost-estimating  organizations  and  program  offices  believe 
Are  Considered  Inadequate  that  cost-estimating  resources  are  inadequate  to  do  a  good  job  of 

accurately  predicting  costs.  They  believe  that  their  cost-estimating 
resources  have  atrophied  over  the  years  because  of  previous  downsizing 
of  the  workforce,  making  resources  such  as  staff  and  data  inadequate. 

As  noted  earlier,  there  was  a  belief  within  the  government  that  cost 
savings  could  be  achieved  under  acquisition  reform  initiatives  by  reducing 
technical  staff,  including  cost  estimators,  since  the  government  would  be 
relying  more  on  commercial-based  solutions  to  achieve  desired 
capabilities.  According  to  one  Air  Force  cost-estimating  official  we  spoke 
with,  this  led  to  a  decline  in  the  number  of  Air  Force  cost  estimators  from 
680  to  280.  High-grade  positions  and  specialty  cost-estimating  job  codes 
were  eliminated,  abolishing  an  official  cost-estimating  career  path,  and 
subordinating  cost  estimating  as  an  additional  duty.  In  the  process, 
according  to  this  same  Air  Force  official,  many  military  and  civilian  cost¬ 
estimating  personnel  left  the  cost-estimating  field,  and  the  Air  Force  lost 
some  of  its  best  and  brightest  cost  estimators. 

Information  we  obtained  from  space  program  offices  and  cost-estimating 
organizations  is  consistent  with  the  assertion  of  a  lack  of  requisite 
resources.  Eight  of  13  cost-estimating  organizations  and  program  offices 
we  informally  surveyed  believe  the  number  of  cost  estimators  is 
inadequate.  Furthermore,  some  of  these  same  organizations  believe  that 
cost  estimation  is  not  a  respected  career  field  within  the  Air  Force,  and 
more  specifically,  that  Air  Force  cost  estimators  are  not  encouraged,  nor 
do  they  have  opportunities  for  promotion  or  advancement.  Regarding  the 
recognition  and  career  paths  for  cost  estimators,  our  data  showed  that 
only  3  of  12  organizations  agreed  that  previous  cost  estimators  had  moved 
on  to  positions  of  equal  or  higher  responsibility.  Further,  only  4  of 
12  agreed  that  people  ask  to  become  cost  estimators. 

The  belief  that  cost-estimating  skills  have  been  depleted  has  been  echoed 
in  other  DOD  and  GAO  studies.  According  to  the  Young  Panel  report, 
government  capabilities  to  lead  and  manage  the  acquisition  process  have 
seriously  eroded,  in  part  because  of  actions  taken  in  the  acquisition  reform 
environment  of  the  1990s.  This  has  extended  to  cost  estimating.  During 
our  2006  review  of  program  management,  we  surveyed  DOD’s  major 
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weapon  system  program  managers  and  interviewed  program  executive 
officers  who  similarly  pointed  to  critical  skill  shortages  for  staff  that 
support  them,  including  cost  estimators.  Other  skill  gaps  identified 
included  systems  engineering,  program  management,  and  software 
development.  We  continue  to  observe  these  deficiencies  in  our  more 
recent  reviews  of  the  space  acquisition  workforce.11 

Because  of  the  decline  in  in-house  cost-estimating  resources,  space 
program  offices  and  Air  Force  cost-estimating  organizations  are  now  more 
dependent  on  support  contractors.  Ten  of  13  cost-estimating  organizations 
and  program  offices  have  more  contractor  personnel  preparing  cost 
estimates  than  government  personnel.  At  11  space  program  offices, 
contractors  account  for  64  percent  of  cost-estimating  personnel.  Support 
contractor  personnel  generally  prepare  cost  estimates,  while  government 
personnel  provide  oversight,  guidance,  and  review  of  the  cost-estimating 
work.  By  contrast,  the  CAIG  had  made  a  determination  that  cost 
estimating  is  too  important  of  a  function  to  place  in  the  hands  of  support 
contractors,  and  assigns  only  government  personnel  to  develop  cost 
estimates. 

Reliance  on  support  contractors  raises  questions  from  the  cost-estimating 
community  about  whether  numbers  and  qualifications  of  government 
personnel  are  sufficient  to  provide  oversight  of  and  insight  into  contractor 
cost  estimates.  A  senior  CAIG  official  involved  with  estimating  for  space 
acquisition  programs,  for  example,  suggested  that  reliance  on  support 
contractors  is  a  problem  if  the  government  cannot  evaluate  how  good  a 
cost  estimate  is  or  lacks  the  ability  to  track  it.  Two  studies  have  also  raised 
the  concern  that  relying  on  support  contractors  makes  it  more  difficult  to 
retain  institutional  knowledge  and  instill  accountability.  Further,  in  the 
most  recent  defense  authorization  act,  Congress  is  requiring  DOD  to  make 
it  a  goal  that  within  5  years  certain  critical  acquisition  functions,  including 
cost  estimating,  be  performed  by  properly  qualified  DOD  employees,  and 
that  in  developing  a  comprehensive  strategy  for  supporting  the  program 


11  GAO,  Defense  Space  Activities:  Management  Actions  Are  Needed  to  Better  Identify, 
Track,  and  Train  Air  Force  Space  Personnel,  GAO-06-908  (Washington,  D.C.:  Sept.  21, 
2006),  and  Defense  Acquisitions:  DOD  Needs  to  Establish  an  Implementing  Directive  to 
Publish  Information  and  Take  Actions  to  Improve  DOD  Information  on  Critical 
Acquisition  Positions,  GAO-06-987R  (Washington,  D.C.:  Sept.  8,  2006). 
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manager  role,  DOD  address  improved  resources  and  support  such  as  cost¬ 
estimating  expertise.12 

A  second  resource  gap  hampering  cost  estimating  is  the  lack  of  reliable 
technical  source  data.  Officials  we  spoke  with  believe  that  cost  estimation 
data  and  databases  from  which  to  base  cost  estimates  are  incomplete, 
insufficient,  and  outdated.  They  cite  a  lack  of  reliable  historical  and 
current  cost,  technical,  and  programmatic  data  and  expressed  concerns 
that  available  cost,  schedule,  technical,  and  risk  data  are  not  similar  to  the 
systems  they  are  developing  cost  estimates  for.  In  addition,  some 
expressed  concerns  that  relevant  classified  and  proprietary  commercial 
data  may  exist  but  are  not  usually  available  to  the  cost-estimating 
community  working  on  unclassified  programs.  Some  believe  that  Air 
Force  cost  estimators  need  to  be  able  to  use  all  relevant  data,  including 
those  contained  in  NRO  cost  databases,  since  the  agency  builds  highly 
complex,  classified  satellites  in  comparable  time  and  at  comparable  costs 
per  pound. 


Successful 
Organization 
Approaches  That 
Better  Support  Cost 
Estimating 


Over  the  past  decade,  GAO  has  examined  successful  organizations  in  the 
commercial  sector  to  identify  best  practices  that  can  be  applied  to  weapon 
system  acquisitions.  This  work  has  identified  a  number  of  practices  that 
better  support  cost  estimating  than  DOD  does.  For  instance,  unlike  most 
space  programs  we  have  reviewed,  the  successful  organizations  we  have 
studied  extensively  researched  and  defined  requirements  before  program 
start  to  ensure  that  they  are  achievable,  given  available  resources.  They  do 
not  define  requirements  after  starting  programs.  They  also  ensure 
technologies  are  mature,  that  is,  proven  to  work  as  intended,  and  assign 
more  ambitious  efforts  to  corporate  research  departments  until  they  are 
ready  to  be  added  to  future  increments.  In  addition,  these  organizations 
use  systems  engineering  to  close  gaps  between  resources  and 
requirements  before  launching  the  development  process.  Taken  together, 
these  practices  help  ensure  that  there  is  little  guessing  in  how  long  or  how 
many  dollars  it  will  take  to  achieve  an  intended  capability.  Moreover, 
within  the  organizations  we  studied,  decisions  to  start  programs  are  made 
through  long-term  strategic  planning  and  prioritizing.  As  a  result, 
competition  for  funding  is  minimized,  and  programs  themselves  do  not 
have  incentives  to  present  low  estimates. 


12  John  Warner  National  Defense  Authorization  Act  for  Fiscal  Year  2007,  Pub.  L.  No.  109-364 
§§  820,  853  (2006). 


Page  22 


GAO-07-96  Space  Acquisitions 


The  successful  organizations  we  have  studied  have  taken  additional  steps 
to  ensure  cost  estimates  are  complete  and  accurate  that  DOD  has  not.  For 
instance,  they  hold  program  managers  accountable  for  their  estimates  and 
require  program  managers  to  stay  with  a  project  to  its  end.  At  the  same 
time,  they  develop  common  templates  and  tools  to  support  data  gathering 
and  analysis  and  maintain  databases  of  historical  cost,  schedule,  quality, 
test,  and  performance  data.  Cost  estimates  themselves  are  continually 
monitored  and  regularly  updated  through  a  series  of  numerous  gates  or 
milestone  decisions  that  demand  programs  assess  readiness  and  remaining 
risk  within  key  sectors  of  the  program  as  well  as  overall  cost  and  schedule 
issues. 

Senior  leaders  within  these  organizations  also  actively  encourage  program 
managers  to  share  bad  news  about  their  programs  and  spend  a  great  deal 
of  time  breaking  down  stovepipes  and  other  barriers  to  sharing 
information.  More  important,  they  commit  to  fully  funding  programs  and 
adhere  to  those  commitments.  Commonly,  the  organizations  we  studied 
have  centralized  cost  estimators  and  other  technical  and  business  experts 
so  that  there  is  more  effective  deployment  of  technical  and  business  skills 
while  at  the  same  time  ensuring  some  measure  of  independence.  Within 
DOD,  the  CAIG  is  a  good  example  of  this.  Its  cost  estimates  are  produced 
by  civilian  government  personnel  (the  sole  military  space  cost  estimating 
position  will  convert  to  a  civilian  position  later  on  this  year  when  the 
military  cost  estimator  retires),  to  ensure  long-term  institutional 
knowledge  and  limit  the  effects  of  staff  turnover  that  commonly  occur 
with  military  personnel.  Although  the  CAIG  uses  support  contractors  for 
conducting  studies,  it  does  not  allow  cost  estimates  to  be  developed  by 
contractors.  The  CAIG  takes  this  approach  because  it  considers  cost 
estimating  to  be  a  core  function  and  therefore  too  important  to  contract 
out.  The  Naval  Air  Systems  Command’s  Cost  Analysis  Division  is  also 
considered  a  model  by  some  in  the  cost-estimating  community  because  of 
its  organizational  structure  and  leadership  support.  It  is  a  centralized  cost 
department  that  provides  support  to  multiple  program  offices.  The 
department  is  headed  by  a  senior  executive-level  manager,  and  various 
branches  within  the  department  are  headed  by  GS-15-leveI  managers. 
Analysts  are  somewhat  independent  of  the  program  offices,  as  their 
supervisors  are  within  the  engineering  department.  This  cost  department 
has  strong  support  from  its  leadership,  and  this  support  has  helped  it  hire 
the  number  of  analysts  and  receive  the  resources  it  needs.  However, 
another  official  pointed  out  that  this  cost  department  is  not  completely 
independent  from  the  acquisition  chain  of  command,  since  it  receives 
funding  from  the  program  offices  to  conduct  the  cost  estimates. 
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GAO  has  made  recommendations  to  DOD  to  adopt  best  practices  we  have 
identified  that  would  strengthen  program  management  DOD-wide. 
Congress  also  recently  directed  DOD  to  develop  a  strategy  to  enhance 
program  manager  empowerment  and  accountability,  agreeing  with  GAO’s 
assessment  that  DOD  has  consistently  failed  to  give  program  managers  the 
authority  that  they  need  to  successfully  execute  acquisition  programs  and, 
as  a  result,  is  unable  to  hold  them  accountable.13  GAO  has  also  made 
recommendations  to  the  Air  Force  to  better  position  its  space  programs 
for  success.  In  response,  the  Air  Force  has  restructured  its 
Transformational  Satellite  Communications  System  (TSAT)  to  ensure  that 
the  program  incorporates  technologies  that  have  been  proven  to  work  as 
intended,  and  it  has  deferred  more  ambitious  efforts  to  the  science  and 
technology  community.  It  has  committed  to  do  the  same  on  other 
programs.  If  effectively  implemented,  such  actions  would,  in  turn, 
significantly  enhance  the  ability  of  independent  estimators  to  forecast 
costs.  However,  we  have  testified  that  DOD  faces  a  number  of  challenges 
and  impediments  in  its  effort  to  instill  this  approach.  It  needs  significant 
shifts  in  thinking  about  how  space  systems  should  be  developed,  changes 
in  incentives  and  perceptions;  and  further  policy  and  process  changes. 

And  such  changes  will  need  to  be  made  within  a  larger  acquisition 
environment  that  still  encourages  a  competition  for  funding  and 
consequently  pressures  programs  to  view  success  as  the  ability  to  secure 
the  next  installment  rather  than  the  end  goal  of  delivering  the  capabilities 
when  and  as  promised. 

The  Air  Force  has  also  been  taking  actions  to  make  specific  improvements 
to  cost  estimating  for  space  programs.  In  the  case  of  TSAT,  program 
officials  said  they  are  updating  the  program’s  planning  cost  estimate  on  an 
annual  basis.  Furthermore,  according  to  one  CAIG  official,  some  program 
offices  have  recently  been  using  the  CAIG’s  independent  cost  estimates. 
Both  the  SBIRS  High  and  NPOESS  program  offices  are  developing  their 
budgets  based  on  the  CAIG  independent  estimates  that  support  the 
certification  process  for  the  programs’  most  recent  Nunn-McCurdy 
breaches.  Further,  DOD  and  Air  Force  cost  estimators  we  spoke  to 
recognize  that  amendments  made  to  the  Nunn-McCurdy  law  by  the  2006 
Defense  Authorization  Act  may  increase  realism  in  establishing  initial  cost 
estimates.  As  part  of  the  revisions,  DOD  is  barred  from  changing  its 


13  John  Warner  National  Defense  Authorization  Act  for  Fiscal  Year  2007,  Pub.  L.  No. 
109-364,  §  853  (2006),  and  accompanying  conference  report,  H.R.  Rep.  No.  109-702,  pages 
784-785. 
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original  baseline  cost  estimate  for  a  program  until  after  it  has  breached 
certain  Nunn-McCurdy  thresholds  that  require  a  certification  and 
assessment  of  the  program,  and  DOD  must  report  the  baseline  changes  to 
Congress. 

The  Air  Force  has  also  committed  to  strengthening  its  cost-estimating 
capabilities  in  terms  of  people,  methodologies,  and  tools.  For  instance, 

50  new  cost  estimators  have  recently  been  authorized  to  the  AFCAA,  some 
of  whom  may  be  detailed  to  the  Space  and  Missile  Systems  Center.  Finally, 
key  players  within  the  DOD  space  cost-estimating  community  are  meeting 
on  a  regular  basis  to  discuss  issues,  review  recent  findings  from  GAO  and 
other  groups,  and  explore  lessons  learned  and  potential  ideas  for 
improvement. 


Conclusions 


Costs  for  DOD  space  acquisitions  over  the  past  several  decades  have 
consistently  been  underestimated — sometimes  by  billions  of  dollars.  For 
the  most  part,  this  has  not  been  caused  by  poor  cost  estimating  itself,  but 
rather  the  tendency  to  start  programs  before  knowing  whether 
requirements  can  be  achieved  within  available  resources.  In  fact,  with  so 
many  unknowns  about  what  could  be  achieved,  how,  and  when,  even  the 
most  rigorous  independent  cost  estimate  could  have  been  off  by  a 
significant  margin.  Nevertheless,  in  the  past,  the  Air  Force  has 
exacerbated  acquisition  problems  by  not  relying  on  independent  cost 
estimates  and  failing  to  encourage  more  realism  in  program  planning  and 
budgeting.  Moreover,  even  after  the  Air  Force  embraced  independent  cost 
estimating  in  its  acquisition  policy  for  space,  it  did  not  facilitate  better 
estimating  by  according  the  cost-estimating  community  with  the 
organizational  clout,  support,  and  guidance  the  Air  Force  believes  are 
needed  to  ensure  the  community’s  analyses  are  used.  On  a  positive  note, 
the  Air  Force  has  committed  to  addressing  some  of  the  root  causes  behind 
cost  growth,  principally  by  accumulating  more  knowledge  about 
technologies  before  starting  new  programs.  Though  adopting  this 
approach  will  be  challenging  without  larger  DOD  acquisition,  funding,  and 
requirement-setting  reforms,  the  Air  Force  can  facilitate  better  planning 
and  funding  approaches  by  aligning  resources  and  policy  to  support 
improved  cost-estimating  capability  and  by  following  through  on  its 
commitment  to  use  independent  estimates. 
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Recommendations  for 
Executive  Action 


We  recommend  that  the  Secretary  of  Defense  direct  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics  or  the  Secretary  of 
the  Air  Force,  as  appropriate,  to  take  the  following  actions: 

1.  To  increase  accountability  and  transparency  of  decisions  in  space 
programs  where  an  independent  estimate  produced  by  the  CAIG  or 
AFCAA  is  not  chosen,  require  officials  involved  in  milestone  decisions 
to  document  and  justify  the  reasons  for  their  choice  and  the 
differences  between  the  program  cost  estimate  and  the  independent 
cost  estimate. 

2.  To  better  ensure  investment  decisions  for  space  programs  are 
knowledge-based,  instill  processes  and  tools  necessary  to  ensure 
lessons  learned  are  incorporated  into  future  estimates.  This  could 
include 

•  conducting  postmortem  reviews  of  past  space  program  cost 
estimates  (program  office  and  independent  cost  estimates)  to 
measure  cost-estimating  effectiveness  and  to  track  and  record  cost¬ 
estimating  mistakes; 

•  developing  a  centralized  cost-estimating  database  that  provides 
realistic  and  credible  data  to  cost  estimators; 

•  establishing  protocols  by  which  cost  estimators  working  with  the 
National  Reconnaissance  Office  can  share  data  with  the  DOD  space 
cost-estimating  community  while  still  maintaining  appropriate 
security  over  classified  data;  and 

•  ensuring  estimates  are  updated  as  major  events  occur  within  a 
program  that  could  have  a  material  impact  on  cost,  such  as  budget 
reductions,  integration  problems,  hardware/software  quality 
problems,  and  so  forth. 

3.  To  optimize  analysis  and  collaboration  within  the  space  cost¬ 
estimating  community,  clearly  articulate  the  roles  and  responsibilities 
of  the  various  Air  Force  cost-estimating  organizations,  and  ensure  that 
space  system  cost  estimators  are  organized  so  that  the  Air  Force  can 
gain  the  most  from  their  knowledge  and  expertise.  In  taking  these 
actions  for  programs  for  which  no  independent  estimate  is  developed 
by  the  CAIG,  consider  assigning  AFCAA  the  responsibility  for  the 
development  of  independent  cost  estimates  for  space  system 
acquisitions,  since  it  is  outside  of  the  acquisition  chain  of  command 
and  therefore  likely  to  be  unbiased  and  not  pressured  to  produce 
optimistic  estimates. 


Page  26 


GAO-07-96  Space  Acquisitions 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  DOD 
concurred  with  the  overall  findings  in  our  report  and  provided  technical 
comments,  which  have  been  incorporated  where  appropriate.  DOD  also 
concurred  with  two  of  our  recommendations  and  partially  concurred  with 
one. 

DOD  concurred  with  our  recommendation  to  instill  processes  and  tools 
necessary  to  ensure  lessons  learned  are  incorporated  into  future 
estimates.  DOD  stated  it  was  already  taking  actions  to  address  our 
recommendations.  For  example,  the  CAIG  has  established  a  process 
whereby  key  members  of  the  national  security  space  cost  analysis 
community  meet  to  discuss  and  evaluate  outcomes  following  ACAT  I 
space  program  milestone  reviews  or  key  decision  point  Defense 
Acquisition  Board-level  reviews,  to  provide  visibility  to  other  members  of 
the  community  on  how  the  CAIG  approaches  independent  cost  estimate 
development  and  to  give  the  community  an  opportunity  to  provide 
feedback  to  the  CAIG  on  how  to  improve  its  processes.  DOD  stated  that 
the  CAIG  will  work  in  the  future  to  incorporate  peer  reviews  of  the 
program  office  estimates  within  this  existing  framework.  DOD  also 
concurred  with  our  recommendation  to  develop  a  centralized  cost¬ 
estimating  database,  and  stated  that  several  groups  within  the  space  cost¬ 
estimating  community  have  been  working  to  develop  a  database  of 
historical  space  program  costs  available  to  the  community  as  a  whole,  and 
has  also  reestablished  a  common  space  program  work  breakdown 
structure  that  supports  the  various  estimating  methodologies  employed  by 
the  space  cost  community.  Through  the  common  database  development 
process,  the  community  is  working  to  make  historical  program  cost  data 
as  widely  available  as  possible.  DOD  also  agreed  with  our 
recommendation  to  update  cost  estimates  as  major  events  occur  within  a 
program,  as  long  as  they  are  program  and  program  phase  dependent. 
Finally,  DOD  concurred  with  our  recommendation  to  clearly  articulate  the 
roles  and  responsibilities  of  the  various  cost-estimating  organizations. 

DOD  stated  that  the  Air  Force  is  currently  updating  its  policy  directive  to 
further  clarify  the  roles  and  responsibilities  of  the  space  cost  analysis 
organizations  to  optimize  analysis  and  collaborations,  thus  making  the 
best  use  of  the  limited  number  of  qualified  and  experienced  space  program 
cost  analysts.  We  agree  that  these  actions  are  steps  in  the  right  direction 
and  that  they  will  strengthen  cost-estimating  capabilities  and  improve 
space  program  cost  estimates. 

DOD  partially  concurred  with  our  recommendation  to  require  officials 
involved  in  milestone  decisions  to  document  and  justify  the  reasons  for 
their  cost  estimate  choice  and  the  differences  between  the  program  cost 


Page  27 


GAO-07-96  Space  Acquisitions 


estimate  and  the  independent  cost  estimate.  In  commenting  on  this 
recommendation,  DOD  stated  that  the  complex  decision  to  det  ermine 
which  cost  figure  to  use  as  basis  for  funding  and  to  evaluate  future 
program  performance  must  weigh  many  competing  factors  that  are  often 
qualitative  in  nature.  It  further  stated  that  the  decision  is  the  milestone 
decision  authority’s  alone,  and  that  documenting  the  explicit  justification 
will  reduce  the  milestone  decision  authority’s  future  decision-making 
flexibility.  We  do  not  see  how  documenting  the  explicit  justification  will 
significantly  reduce  the  milestone  decision  authority’s  future  decision¬ 
making  flexibility.  While  we  recognize  the  value  of  decision-making 
flexibility  and  the  role  that  judgment  must  play  in  such  decisions,  we  also 
believe  that  the  basis  for  the  decisions  should  withstand  review, 
particularly  after  the  person  who  made  the  decision  has  left  office.  We  also 
believe  that  the  greater  transparency  of  cost-estimating  decisions  that  a 
documented  justification  provides  is  needed,  particularly  in  light  of  the 
poor  foundation  of  choices  made  in  the  past  on  space  programs. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees  and  the  Secretaries  of  Defense  and  the  Air  Force.  We  will  also 
provide  copies  to  others  on  request.  In  addition,  this  report  will  be 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  call  me  at  (202)  512-4841  (chaplainc@gao.gov).  Contact  points  for 
our  Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  GAO  staff  who  made  major  contributions  to 
this  report  are  listed  in  appendix  VI. 


Cristina  T.  Chaplain 

Acting  Director,  Acquisition  and  Sourcing  Management 
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Appendix  I:  Scope  and  Methodology 


The  Chairman  and  the  Ranking  Member,  Subcommittee  on  Strategic 
Forces,  House  Committee  on  Armed  Services,  requested  that  we  examine 
(1)  in  what  areas  space  system  acquisitions  cost  estimates  have  been 
unrealistic  and  (2)  what  incentives  and  pressures  have  contributed  to  the 
quality  and  usefulness  of  cost  estimates  for  space  system  acquisitions. 

To  determine  whether  cost  estimates  for  space  system  acquisitions  have 
been  realistic,  we  used  a  case  study  methodology.  We  selected  six  ongoing 
Air  Force  space  system  acquisitions.  We  selected  these  acquisitions 
because  they  were  far  enough  along  in  their  acquisition  cycles  for  us  to  be 
able  to  observe  changes  in  the  programs  since  their  initial  cost  estimates 
were  developed.  The  six  space  system  acquisitions  are  the  Advanced 
Extremely  High  Frequency  Satellites,  the  Evolved  Expendable  Launch 
Vehicle,  the  Global  Positioning  System  IIF,  the  National  Polar-orbiting 
Operational  Environmental  Satellite  System,  the  Space  Based  Infrared 
System  High,  and  the  Wideband  Gapfiller  Satellites.  For  each  of  the  case 
studies,  we  met  with  the  program  office  representatives  at  the  Air  Force’s 
Space  and  Missile  Systems  Center  and  at  the  program’s  prime  contractors. 
We  also  obtained  program  cost  and  other  program  documentation  to 
determine  how  the  cost  estimates  were  formulated  and  on  what  basis  they 
were  formulated. 

To  determine  what  incentives  and  pressures  contributed  to  the  quality  and 
usefulness  of  cost  estimates  for  space  system  acquisitions,  we  examined 
Department  of  Defense  (DOD)  and  Air  Force  policies  for  developing  and 
updating  cost  estimates  for  space  programs.  We  also  used  a  data  collection 
instrument  to  obtain  information  on  cost-estimating  practices  and 
resources  within  the  Air  Force  Cost  Analysis  Agency,  at  the  Space  and 
Missile  Systems  Center,  and  at  the  space  program  offices.  We  conducted 
interviews  with  the  Office  of  the  Secretary  of  Defense’s  Cost  Analysis 
Improvement  Group,  the  Air  Force  Cost  Analysis  Agency,  and  the  Air 
Force  Space  and  Missile  Systems  Center’s  Cost  Center.  On  the  basis  of  the 
results  of  the  data  collection  instruments  and  interviews,  we  obtained 
information  on  the  organizational  alignment  of  cost-estimating 
organizations,  including  roles  and  responsibilities,  as  well  as  concerns 
over  the  current  cost-estimating  policies  and  practices. 

We  also  relied  on  our  previous  best  practice  studies,  which  have  examined 
pressures  and  incentives  affecting  space  system  acquisition  programs,  the 
optimal  levels  of  knowledge  needed  to  successfully  execute  programs,  and 
complementary  management  practices  and  processes  that  have  helped 
commercial  and  DOD  programs  to  reduce  costs  and  cycle  time.  Moreover, 
we  reviewed  studies  from  the  Defense  Science  Board,  the  DOD  Inspector 
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General,  IBM,  and  others  on  space  system  acquisition  and  cost-estimating 
issues. 

Finally,  we  discussed  the  results  of  our  work  and  our  observations  with  an 
expert  panel  made  up  of  representatives  from  the  DOD  space  cost¬ 
estimating  community. 

We  conducted  our  review  between  August  2005  and  October  2006  in 
accordance  with  generally  accepted  government  auditing  standards. 
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An  acquisition  program  is  categorized  based  on  dollar  value  and  milestone 
decision  authority  special  interest.  Table  5  contains  the  description  and 
decision  authority  for  acquisition  categories  ID  and  IC. 


Table  5:  DOD  Acquisition  Categories  and  Decision  Authorities 

Acquisition  category  (ACAT) 

Dollar  value 

Milestone  decision  authority 

AC  AT  ID 

For  designated  major  defense  acquisition 
programs  (special  interest  based  on 
technological  complexity,  congressional 
interest,  large  commitment  of  resources, 
critical  role  in  achieving  a  capability,  or  a 
joint  program) 

Research,  development,  test,  and 
evaluation  >  $365  million 

Procurement  >  $2.19  billion 

Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics 

ACAT  IC 

For  major  defense  acquisition  programs  not 
designated  as  ACAT  ID 

Research,  development,  test,  and 
evaluation  >  $365  million 

Procurement  >  $2.19  billion 

Head  of  DOD  component  or,  if  delegated, 
DOD  component  or  service  acquisition 
executive 

Source:  DOD  Instruction  5000.2,  Enclosure  2,  which  also  lists  other  acquisition  categories. 


Note:  Dollar  values  are  fiscal  year  2000  constant  dollars. 
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Table  6  highlights  major  areas  where  program  officials  were  too  optimistic 
in  their  assumptions  for  the  six  space  system  acquisitions  we  examined — 
the  Advanced  Extremely  High  Frequency  (AEHF)  Satellites,  the  Evolved 
Expendable  Launch  Vehicle  (EELV),  the  Global  Positioning  System  (GPS) 
IIF,  the  National  Polar-orbiting  Operational  Environmental  Satellite 
System  (NPOESS),  the  Space  Based  Infrared  System  (SBIRS)  High,  and 
the  Wideband  Gapfiller  Satellites  (WGS). 


Table  6:  Examples  of  Optimistic  Assumptions 

Space  program  affected 

Examples 

Assumed  industrial  base  would  remain  constant  and  available 

EELV 

The  original  contracting  concept  was  for  the  Air  Force  to  piggyback  on  the  launch  demand  anticipated  to 
be  generated  by  the  commercial  sector.  However,  the  commercial  demand  never  materialized,  and  the 
government  had  to  take  on  an  additional  cost  burden.  In  addition,  the  cost  for  launch  services  increased 
because  fixed  infrastructure  costs  are  being  spread  over  15  launches  a  year  instead  of  the  original 
expectation  of  75  launches  a  year. 

GPS  IIF 

A  deteriorating  manufacturing  base  of  contractors  and  subcontractors  caused  the  prime  contractor  to 
move  the  design  team  from  Seal  Beach,  California,  to  Anaheim,  California,  in  2001.  Additional  moves 
occurred  as  the  prime  contractor  consolidated  development  facilities  to  remain  competitive.  For  each 
move,  the  prime  contractor  lost  valuable  workers,  causing  inefficiencies  in  the  program.  In  addition,  the 
contractor  took  additional  cost-cutting  measures  that  reduced  quality. 

NPOESS 

A  long  production  phase  on  this  program  increases  the  probability  for  parts  obsolescence.  Over  70 
percent  of  the  value  added  to  the  program  is  from  the  supply  base,  and  some  critical  parts  that  are 
unique  to  the  program  are  produced  by  relatively  small  companies.  In  addition,  workers  required  to  have 
specialized  skills  must  be  United  States  citizens  to  obtain  security  clearances.  The  labor  pool  has  to 
produce  these  specialized  skills  because  degree  programs  currently  do  not  produce  them. 

SBIRS  High 

Consolidation  within  the  supplier  base  has  adversely  affected  the  program.  When  suppliers  merged, 
costs  increased  for  supplier  technical  assistance,  product  rework,  and  hardware  qualifications.  In 
addition,  unforeseen  costs  resulted  when  production  processes  and  materials  were  changed  and 
facilities  and  personnel  were  relocated. 

WGS 

At  the  time  of  contract  award,  the  satellite  industry  was  flourishing  with  commercial  satellite  orders,  and 
the  contractor  anticipated  a  large  market.  However,  when  the  installation  of  optical  fiber  communication 
lines  became  widespread,  many  of  the  commercial  initiatives  involving  proposed  space  systems  did  not 
materialize.  The  government  had  planned  to  gain  leverage  from  the  design  work  of  commercial 
contractors  but  ended  up  having  to  pay  for  design  efforts.  In  addition,  because  of  the  reduction  of  the 
number  of  contracts  awarded,  small  subcontractors  started  to  consolidate.  Specialized  parts  became 
obsolete,  and  the  Air  Force  was  no  longer  considered  a  high-priority  customer. 

Assumed  technology  would  be  mature  enough  when  needed 

AEHF 

AEHF  faced  several  technology  maturity  problems  including  developing  a  digital  processing  system  that 
would  support  10  times  the  capacity  of  Milstar  medium  data  rate  without  self-interference,  and  using 
phased  array  antennas  at  extremely  high  frequencies,  which  had  never  been  done  before.  In  addition, 
the  change  from  a  physical  to  an  electronic  process  for  crypto  re-keys  was  not  expected  at  the  start  of 
the  AEHF.  The  predecessor  program  to  AEHF  was  Milstar,  which  required  approximately  2,400  crypto 
re-keys  per  month,  which  could  be  done  physically.  Regarding  AEHF  proposed  capabilities,  the  number 
of  crypto  re-keys  is  approximately  100,000,  which  is  too  large  for  a  physical  process  and  must  be  done 
electronically.  Changing  the  way  the  re-keys  were  done  called  for  a  revolutionary  change  in  the  process 
and  led  to  unexpected  cost  and  schedule  growth. 
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Space  program  affected 

Examples 

GPS  IIF 

The  cost  estimate  was  built  on  the  assumption  that  the  military  code  being  developed  in  the  program 
would  fit  on  one  chip.  However,  once  development  started,  there  were  interface  issues,  and  the 
subcontractor  had  to  move  to  a  two-chip  design,  which  added  cost  growth  to  the  program.  In  addition, 
the  problem  took  8  months  to  solve. 

NPOESS 

DOD  and  the  Department  of  Commerce  committed  funds  for  the  development  and  production  of  the 
satellites  before  the  design  was  proven  and  before  the  technology  was  mature.  At  program  initiation, 
only  1  of  14  critical  technologies  was  mature,  and  some  technology  levels  have  been  assessed 
downward.  For  example,  the  1394  Bus  Technology  Readiness  Level  (TRL)  was  changed  from  5  to 

4  after  the  contractor  added  more  verification  testing. 

SBIRS  High 

In  2003,  GAO  reported  that  three  critical  technologies— the  infrared  sensor,  thermal  management,  and 
onboard  processor — were  now  mature.  When  the  program  began,  in  1996,  none  of  its  critical 
technologies  was  mature. 

WGS 

The  X-band  phased  array  antennas  and  the  array  power  chips  were  the  most  difficult  technologies  to 
mature,  because  these  state-of-the-art  elements  generated  too  much  heat,  which  is  very  difficult  to 
remove  in  outer  space,  so  they  had  to  be  redesigned. 

Assumed  Total  System  Performance  Responsibility  (TSPR)  would  reduce  costs  and  schedule 

EELV 

The  EELV  program  office  entered  into  a  TSPR  contract  that  does  not  require  the  contractor  to  deliver 
cost  or  earned  value  management  data.  The  program  office  stated  that  TSPR  gave  too  many 
responsibilities  to  the  contractor  and  not  enough  to  the  government. 

GPS  IIF 

The  contract  that  was  awarded  during  acquisition  reform  efforts  of  the  late  1990s  adopted  the  TSPR 
approach.  Under  TSPR,  there  was  limited  oversight  of  the  contractor,  and  this  contributed  to  relaxed 
specifications  and  inspections  on  commercial  practices,  loss  of  quality  in  the  manufacturing  process,  and 
poor-quality  parts  that  caused  test  failures,  unexpected  redesigns,  and  the  late  delivery  of  parts. 

NPOESS 

The  NPOESS  prime  contractor  has  a  Shared  System  Performance  Responsibility  (SSPR),  which  was  an 
outgrowth  of  TSPR.  The  SSPR  arrangement  relegates  the  government’s  role  as  a  participant  in 
contractor  Integrated  Product  Team  meetings.  In  addition,  the  program  is  managed  by  officials  from 
three  separate  government  agencies.  DOD  and  Department  of  Commerce  share  the  cost  of  funding  the 
development  of  NPOESS,  while  NASA  provides  funding  for  specific  technologies  and  studies.  Difficulties 
have  arisen  with  the  tri-service  approach  to  managing  NPOESS,  including  ensuring  NPOESS  follows 
DOD’s  acquisition  process,  but  Commerce,  which  has  control  over  the  program,  has  no  authority  over 
the  DOD  process;  each  agency  is  driven  by  different  program  objectives  (i.e. ,  military,  civilian,  science); 
and  NASA  shares  equally  in  managing  the  program  even  though  it  provides  no  funding  for  the 
development. 

SBIRS  High 

When  the  original  contract  was  awarded,  acquisition  reform  efforts  were  being  implemented  and  called 
for  the  use  of  commercial  practices  instead  of  government  standards.  In  order  to  achieve  cost  savings, 
the  SBIRS  program  office  reduced  critical  up-front  systems  engineering  design  practices  and  follow-on 
quality  assurance  inspections  based  on  the  expectation  that  the  contractor  would  perform  these  activities 
with  no  government  oversight.  The  prime  contractor  also  held  the  same  requirements  for  its 
subcontractors  as  a  way  to  keep  costs  down.  This  lack  of  oversight  resulted  in  difficulties  in  determining 
the  root  causes  when  components  began  to  fail  during  testing.  For  example,  there  have  been  latent 
defects  that  required  extensive  corrective  action  and  associated  cost  growth  with  the  software  redesign, 
single  board  computer  halts,  payload  reference  bench  rework,  payload  electromagnetic  interference, 
software  configuration  issues,  propulsion  solder  issues,  and  telescope  foreign  object  damage.  In 
addition,  the  contractor  had  responsibility  to  coordinate  different  agency  needs,  a  responsibility  that 
proved  to  be  difficult  when  trying  to  resolve  hardware  interface  issues. 
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Assumed  savings  from  heritage  systems 

AEHF 

The  program  office  cost  estimators  relied  on  data  from  heritage  systems  to  estimate  AEHF  nonrecurring 
costs.  The  Cost  Analysis  Improvement  Group  (CAIG)  believed  the  estimates  based  on  heritage  data 
were  subjectively  derived  and  therefore  susceptible  to  bias.  For  example,  AEHF  program  officials 
assumed  that  the  nulling  antennas  would  have  the  same  performance  as  those  on  Milstar,  requiring  little 
if  any  development.  In  fact,  because  of  parts  obsolescence,  personnel  turnover,  and  other  issues,  the 
entire  antenna  had  to  be  redesigned  at  nearly  the  same  cost  as  the  first  one.  There  were  similar  beliefs 
that  legacy  processing  technology  could  be  used,  which  turned  out  to  not  be  possible.  Further,  almost  all 
of  the  payload  software  had  to  be  rewritten  to  support  the  new  hardware.  As  a  result,  there  was  much 
less  technology  transfer  from  Milstar  II  to  AEHF,  even  though  the  contractor  was  the  same. 

NPOESS 

NPOESS  payload  development  proposals  relied  heavily  on  leveraging  heritage  satellite  instrument 
technology  development.  The  prime  contractor  and  the  program  office  agreed  there  was  too  much 
optimism  regarding  heritage  sensor  reuse.  For  example,  the  Visible  Infrared  Radiometer  Suite  (VIIRS)  is 
more  powerful  and  complex  and  will  weigh  20  percent  more  than  the  heritage  sensor  that  was  used  to 
base  the  estimate.  In  addition,  the  Conical  Microwave  Imager  Sounder  (CMIS)  is  much  more  complex 
than  the  heritage  sensor,  which  took  more  than  8  years  to  develop.  The  program  office  estimated  a 

4-year  development  schedule  for  CMIS.  The  latest  cost  estimate  for  CMIS  is  now  approximately  five 
times  the  initial  estimate. 

SBIRS  High 

The  original  estimate  for  nonrecurring  engineering  was  significantly  underestimated  based  on  actual 
experience  in  legacy  sensor  development  and  assumed  software  reuse.  As  a  result,  nonrecurring  costs 
should  have  been  two  to  three  times  higher  according  to  historical  data  and  independent  cost  estimators. 

WGS 

Originally,  the  contractor  planned  to  gain  leverage  from  a  commercial  satellite  development  effort — using 
the  same  bus  and  phased  array  antenna.  The  commercial  satellite  development  effort  did  not 
materialize,  leaving  DOD  to  pay  for  infrastructure  and  hardware  design  costs.  This  caused  WGS  costs  to 
increase  and  the  schedule  to  slip. 

Assumed  no  weight  growth  would  occur 

AEHF 

When  the  cost  estimate  was  initially  developed,  satellite  payload  weight  was  assumed  to  be  constant  by 
the  program  office.  When  updating  its  independent  cost  estimate  in  2004,  the  CAIG  found  that  the 
payload  weight  more  than  doubled  between  the  start  of  development  and  critical  design  review.  Weight 
increased  because  of  the  addition  of  phased  array  antennas,  an  antenna  modification,  and  other 
requirements. 

NPOESS 

The  CMIS  sensor  weight  has  almost  doubled  since  the  preliminary  design  review.  As  a  result, 
engineering  change  proposals  were  issued  to  modify  the  spacecraft  to  accept  the  higher  payload  weight. 

SBIRS  High 

Weight  growth  has  occurred  in  the  spacecraft  and  payload.  The  spacecraft  has  experienced  weight 
growth  of  about  59  percent  because  of  the  need  to  lengthen  and  stiffen  the  structure,  add  a  solar  shield 
to  block  sunlight  from  the  payload,  and  add  missing  wire  and  harnessing.  The  geosynchronous  earth 
orbit  (GEO)  payload  has  experienced  nearly  a  44  percent  weight  growth  because  of  integration 
hardware,  pointing,  and  control  assembly. 

WGS 

Problems  with  solar  panel  concentrators  overheating  caused  a  solar  panel  redesign  that  led  to  additional 
weight  growth  in  the  spacecraft  bus. 

Assumed  funding  stream  would  be  sufficient  and  remain  stable 

AEHF 

The  AEHF  program  sustained  a  $100  million  fiscal  year  2002  funding  cut.  The  program  office  reported 
that  the  funding  cut  would  result  in  a  6-month  launch  delay  to  the  first  three  satellites  and  a  delay  in 
meeting  initial  operational  capability.  The  program  had  rapidly  staffed  personnel  to  support  a  warfighter 
need.  The  funding  cut  resulted  in  contractor  program  reductions  to  fit  within  the  revised  fiscal  year  2002 
budget.  In  addition,  DOD  made  a  decision  to  shift  the  acquisition  strategy  from  buying  five  satellites  at 
one  time  to  buying  three  satellites  as  individual  buys,  which  also  caused  costs  to  rise. 
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GPS  IIF 

The  Operational  Control  Segment  portion  of  the  GPS  IIF  program  received  a  $37.7  million  funding  cut  in 
fiscal  year  2005.  Because  of  the  funding  cut,  the  program  delayed  some  of  the  software  efforts  and 
reduced  some  software  requirements. 

NPOESS 

Between  fiscal  years  2004  and  2005,  DOD  reduced  funding  for  the  program  by  about  $65  million. 

However,  funding  was  reduced  $130  million  since  the  Department  of  Commerce  contributes  no  more 
funding  towards  the  program  than  DOD.  The  program  office  determined  that  the  funding  cut  resulted  in 
satellite  launch  delays  ranging  from  5  to  26  months  and  a  cost  increase  of  $391 .2  million. 

SBIRS  High 

A  funding  cut  in  1998-1999  because  of  higher  budget  priorities  caused  a  reduction  in  the  systems 
engineering  staff  and  contributed  to  a  2-year  delay  of  the  geosynchronous  earth  orbit  satellites.  This  cut 
caused  work  activities  to  continually  stop  and  restart  and  drove  the  need  for  interim  solutions  that 
resulted  in  program  instability  and  cost  growth.  It  also  led  to  a  breach  of  the  acquisition  program  baseline 
in  2001 ,  resulting  in  a  change  in  the  procurement  strategy  from  a  single  buy  of  five  satellites  to  two 
separate  buys — one  for  two  satellites  and  the  other  for  three  satellites.  Independent  cost  estimators 
calculated  that  costs  would  double  as  a  result  of  the  change  in  procurement  strategy. 

Assumed  an  aggressive  schedule 

AEHF 

The  first  launch  was  originally  scheduled  for  June  2006,  but  in  response  to  a  potential  gap  in  satellite 
coverage  due  to  the  launch  failure  of  the  third  Milstar  satellite,  DOD  accelerated  the  schedule  by 

18  months,  aiming  for  a  first  launch  in  December  2004.  An  unsolicited  contractor  proposal  stated  that  it 
could  meet  the  accelerated  date,  even  though  all  the  requirements  for  AEHF  were  not  fully  determined. 

As  a  result,  the  program  office  knew  that  the  proposed  schedule  was  overly  optimistic,  but  the  decision 
was  made  at  high  levels  in  DOD  to  award  the  contract.  However,  DOD  did  not  commit  the  funding  to 
support  the  activities  and  manpower  needed  to  design  and  build  the  satellites  more  quickly.  Funding 
issues  further  hampered  development  efforts  and  increased  schedule  delays  and  contributed  to  cost 
increases. 

NPOESS 

When  the  estimate  was  developed,  NPOESS  was  expected  to  be  heavier,  require  more  power,  and  have 
over  twice  as  many  sensors  than  heritage  satellites.  Yet  the  program  office  estimated  that  the  satellites 
would  be  developed,  integrated,  and  tested  in  less  time  than  heritage  satellites.  Independent  cost 
estimators  highlighted  to  the  NPOESS  program  office  that  the  proposed  integration  schedule  was 
unrealistic  when  compared  to  historical  satellite  programs.  Later,  the  CAIG  cautioned  the  program  office 
that  not  only  was  the  system  integration  assembly  and  test  schedule  unrealistic,  but  the  assumptions 
used  to  develop  the  estimate  were  not  credible. 

SBIRS  High 

The  schedule  proposed  in  1996  did  not  allow  sufficient  time  for  geosynchronous  earth  orbit  system 
integration  and  did  not  anticipate  program  design  and  workmanship  flaws,  which  eventually  cost  the 
program  considerable  delays.  In  addition,  the  schedule  was  optimistic  in  regard  to  ground  software 
productivity,  and  time  needed  to  calibrate  and  assess  the  health  of  the  satellite.  There  has  been  almost  a 
3-year  delay  in  the  delivery  of  the  highly  elliptical  orbit  (HEO)  sensors  and  a  6-year  delay  in  the  launch  of 
the  first  GEO  satellite. 

WGS 

The  request  for  proposals  specified  that  the  budget  available  was  $750  million  for  three  satellites  and  the 
ground  control  system  to  be  delivered  within  36  months.  On  the  basis  of  these  requirements,  competing 
contractors  were  asked  to  offer  maximum  capacity,  coverage,  and  connectivity  through  a  contract  that 
would  make  use  of  existing  commercial  practices  and  technologies.  However,  higher  design  complexity 
and  supplier  quality  issues  caused  the  WGS  schedule  to  stretch  to  78  months  for  the  first  expected 
launch.  Historically,  DOD  experienced  between  55  and  79  months  to  develop  satellites  similar  to  WGS, 
so  while  DOD’s  experience  is  within  the  expected  range,  the  original  36-month  schedule  was  unrealistic. 
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Assumed  no  growth  in  requirements 

AEHF 

DOD  awarded  the  contract  for  AEHF  before  the  requirements  were  fully  established  to  fill  the  gap  left  by 
the  Milstar  launch  failure.  As  a  result,  DOD  frequently  and  substantially  altered  requirements  in  the  early 
phases  of  the  program  and  changed  the  system  design.  For  example,  a  new  requirement  increased  the 
need  for  anti-jamming  protection,  which  led  to  a  cost  increase  of  $100  million.  In  addition,  new 
requirements  related  to  training,  support,  and  maintainability  led  to  a  cost  increase  of  $90  million. 

GPS  IIF 

GPS  IIF  was  intended  to  follow  on  the  GPS  II  program,  yet  shortly  after  the  contract  was  awarded,  the 
government  added  the  requirement  for  an  additional  auxiliary  payload.  This  requirement  caused  the 
satellite  design  to  be  larger  than  originally  planned  and,  in  turn,  required  a  larger  launch  vehicle. 
Requirements  for  more  robust  jamming  capability  to  secure  satellite  transmissions  were  also  added. 
Changes  from  a  two-panel  to  a  three-panel  solar  array  design  and  flexible  power  were  necessary  to 
allow  for  more  power  and  thermal  capability  requirements. 

SBIRS  High 

DOD  is  developing  SBIRS  High  to  improve  missile  warning,  missile  defense,  technical  intelligence,  and 
battle-space  characterization.  As  such,  SBIRS  has  many  customers,  including  the  Air  Force,  Army, 
missile  defense,  and  other  agencies,  each  of  which  has  its  own  requirements.  This  has  resulted  in 
complications  in  developing  SBIRS,  due  to  the  fact  that  there  are  19  key  performance  parameters  to 
satisfy,  which  are  about  five  times  more  than  the  typical  DOD  program.  In  addition,  there  are  over 

12,600  requirements  for  the  program  to  address,  and  to  date,  requirements  from  external  users  have  not 
been  fully  defined.  Under  the  TSPR  arrangement,  the  contractor  was  responsible  for  coordinating  these 
requirements.  This  effort  was  challenging  and,  according  to  a  DOD  official,  one  better  suited  for  the 
government  because  all  agencies  were  to  agree  on  requirements.  The  SBIRS  contractor  encountered 
numerous  problems  when  trying  to  resolve  the  interface  issues  among  the  various  agencies.  Moreover, 
the  development  of  interface  control  documents  required  different  certification  requirements  for  each 
agency,  and  the  SBIRS  contractor  had  limited  systems  engineers  to  handle  the  workload.  This  lack  of 
staff  resulted  in  many  requirements  not  flowing  down,  which  led  to  problems  later  on. 

Source:  This  table  is  based  on  conversations  with  program  and  contracting  officials  and  analysis  of  data  they  provided.  In  some  cases, 
we  made  our  own  designations  based  on  our  prior  findings. 
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We  found  examples  from  our  close  examinations  of  the  AEHF,  NPOESS, 
and  SBIRS  High  programs  where  independent  cost  estimates  were  not 
relied  upon  by  program  decision  makers.  Independent  estimates  for  these 
space  system  acquisitions  forecasted  higher  costs  and  lengthier  schedules 
than  program  office  or  service  cost  estimates.  This  appendix  provides 
detailed  information  on  the  differences  between  the  program  office  cost 
estimates  and  the  independent  cost  estimates  for  the  AEHF,  NPOESS,  and 
SBIRS  High  programs. 


AEHF 


In  2004,  AEHF  program  decision  makers  relied  upon  the  program  office 
cost  estimate  rather  than  the  independent  estimate  developed  by  the  CAIG 
to  support  the  production  decision  for  the  AEHF  program.  At  that  time, 
the  AEHF  program  office  estimated  the  system  would  cost  about 
$6  billion.  This  was  based  on  the  assumption  that  AEHF  would  have 
10  times  more  capacity  than  the  predecessor  satellite — Milstar — but  at  half 
the  cost  and  weight.  However,  the  CAIG  concluded  that  the  program  could 
not  deliver  more  data  capacity  at  half  of  the  weight  given  the  state  of 
technology  at  that  time.  In  fact,  the  CAIG  believed  that  in  order  to  get  the 
desired  increase  in  data  rate,  the  weight  would  have  to  increase 
proportionally.  As  a  result,  the  CAIG  estimated  that  AEHF  would  cost 
$8.7  billion,  and  predicted  a  $2.7  billion  cost  overrun  for  the  AEHF 
program.  Table  7  displays  the  differences  between  the  program  office  and 
CAIG  cost  estimates. 


Table  7:  Comparison  of  2004  AEHF  Program  Office  and  Independent  Cost  Estimates 


Millions  of  fiscal  year  2006  dollars 


Independent  cost  estimate 


Program  office 
estimate 

AFCAA 

CAIG 

Difference 

Latest  program 
office  estimate 

$6,015 

a 

$8,688 

44% 

$6,132 

Source:  CAIG  and  GAO  analysis. 

“AFCAA  worked  jointly  with  the  CAIG  to  develop  the  independent  estimate. 


The  CAIG  relied  on  weight  data  from  historical  satellites  to  estimate  the 
cost  of  AEHF  because  it  considers  weight  to  be  the  single  best  cost 
predictor  for  military  satellite  communications.  The  historical  data  from 
the  AEHF  contractor  showed  that  the  weight  had  more  than  doubled  since 
the  program  began  and  the  majority  of  the  weight  growth  was  in  the 
payload.  The  Air  Force  also  used  weight  as  a  cost  predictor,  but  attributed 
the  weight  growth  to  structural  components  rather  than  the  more  costly 
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payload  portion  of  the  satellite.  When  the  CAIG  briefed  the  Air  Force  on 
its  estimate,  the  program  office  disagreed  with  the  CAIG  results,  saying  it 
did  not  see  much  payload  weight  growth  in  the  data  it  analyzed.  The  CAIG 
reported  that  it  used  AEHF  contractor  cost  reports  to  determine  the 
amount  of  weight  growth  for  the  payload,  and  that  these  data  were 
corroborated  by  AEHF  monthly  earned  value  management  data,  which 
showed  cost  overruns  for  the  payload  effort.  As  table  8  shows,  the  payload 
weight  for  the  AEHF  satellite  increased  about  116  percent. 


Table  8:  Historical  AEHF  Weight  Growth 

Milestone 

Date 

Payload  weight  (lbs) 

Percent  growth 

Milestone  1  (A) 

January  1999 

1,694 

n/a 

Milestone  II  (B) 

May  2001 

2,631 

55 

Preliminary  design  review 

August  2001 

3,437 

103 

Critical  design  review 

April  2004 

3,659 

116 

Source:  CAIG. 


The  Air  Force  attributed  AEHF  cost  growth  to  problems  to  the 
cryptographic  portion  of  the  program,  which  is  being  developed  by  the 
National  Security  Agency  (NSA).  AEHF  program  officials  stated  that 
weight  growth  was  consistent  with  that  of  other  space  programs. 
However,  the  CAIG  stated  that  major  cost  growth  was  inevitable  from  the 
start  of  the  AEHF  program  because  historical  data  showed  that  it  was 
possible  to  achieve  a  weight  reduction  or  an  increase  in  data  capacity,  but 
not  both  at  the  same  time. 

In  addition,  the  CAIG  also  stated  that  the  Air  Force  was  optimistic  in 
developing  the  AEHF  schedule  estimate.  During  the  production  decision 
review  in  2004,  the  CAIG  estimated  the  first  satellite  launch  date  to  be 
28  months  longer  than  the  program  office  estimate,  which  the  CAIG 
estimated  to  have  no  more  than  a  1  percent  chance  of  success.  The  CAIG 
also  stated  that  because  of  problems  with  cryptographic  development  and 
reliability  concerns  with  other  technical  aspects  of  the  program,  such  as 
the  phased  array  antenna  and  digital  signal  processing,  the  ambitious 
AEHF  schedule  was  in  jeopardy,  and  the  program  would  not  likely  be 
implemented  as  planned. 

In  February  2006,  the  CAIG  reviewed  the  proposed  revision  to  the  AEHF 
Acquisition  Program  Baseline  (APB).  In  a  memorandum  sent  to  the 
Assistant  Secretary  of  Defense  for  Network  and  Information  Integration, 
the  CAIG  chairman  did  not  concur  with  the  AEHF  draft  APB.  The  CAIG 
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chairman  explained  that  while  the  Air  Force  estimate  included  a 
24  percent  increase  to  the  average  procurement  unit  cost,  which  was  1 
percent  below  the  threshold  for  a  Nunn-McCurdy  certification,  the  CAIG’s 
estimate  prepared  in  December  2004  projected  an  increase  of  over 
100  percent.  Further,  because  of  the  vast  differences  between  the  Air 
Force  and  CAIG  cost  estimates,  the  CAIG  chairman  expressed  concern 
that  Congress  would  perceive  the  revised  APB  as  an  attempt  to  avoid  a 
Nunn-McCurdy  certification. 

There  is  still  risk  for  the  AEHF  program  costs  to  grow.  As  a  result  of 
delays,  AEHF  satellites  have  not  yet  been  through  thermal  vacuum  testing. 
Spacecraft  must  endure  a  wide  range  of  temperatures  associated  with 
liftoff  and  ascent  through  the  atmosphere  and  exposure  to  the  extreme 
temperatures  of  space.  The  thermal  environment  is  generally  considered 
the  most  stressful  operating  environment  for  hardware,  and  electronic 
parts  are  especially  sensitive  to  thermal  conditions.  Problems  such  as 
cracks,  bond  defects,  discoloration,  performance  drift,  coating  damage, 
and  solder-joint  failure  have  typically  occurred.  Thermal  vacuum  testing  is 
used  to  screen  out  components  with  physical  flaws  and  demonstrate  that  a 
device  can  activate  and  operate  in  extreme  and  changing  temperatures. 
Because  thermal  vacuum  testing  provides  the  most  realistic  simulation  of 
flight  conditions,  problems  typically  occur  during  testing.  If  this  occurs  on 
AEHF,  more  delays  and  cost  overruns  are  likely. 


NPOESS 


NPOESS  provides  another  example  of  where  there  were  large  differences 
between  program  office  and  independent  cost  estimates.  In  2003, 
government  decision  makers  relied  on  the  program  office’s  $7.2  billion 
cost  estimate  rather  than  the  $8.8  billion  independent  cost  estimate 
presented  by  the  AFCAA  to  support  the  NPOESS  development  contract 
award.  Program  officials  and  decision  makers  preferred  the  more 
optimistic  assumptions  and  costs  of  the  program  office  estimate,  viewing 
the  independent  estimate  as  too  high.  The  $1.65  billion  difference  between 
the  estimates  is  shown  in  table  9. 
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Table  9:  Comparison  of  2003  NPOESS  Program  Office  and  Independent  Cost 
Estimates 

Millions  of  fiscal  year  2006  dollars 

Independent  cost  estimate 

Program  office 
estimate 

AFCAA  CAIG 

Delta 

Latest  program 
office  estimate 

$7,219 

$8,869 

23% 

$11,400 

Source:  Air  Force  Cost  Analysis  Improvement  Group  briefing,  April  2003. 

Note:  The  program  office  and  the  AFCAA  cost  estimates  were  based  on  a  purchase  of  six  satellites, 
and  the  latest  estimate  is  based  on  a  purchase  of  four  satellites,  with  less  capability  and  a  renewed 
reliance  on  a  European  contribution. 

“The  CAIG  was  not  involved  in  preparing  the  2003  independent  cost  estimate. 

AFCAA  based  its  estimate  on  an  analysis  of  historical  data  from  satellite 
systems  (i.e.,  NASA’s  Aqua  and  Aura  and  DOD’s  Defense  Meteorological 
Satellite  Program  [DMSP]  program)1  independent  software  and  hardware 
models,  and  a  risk  simulation  model  using  input  from  30  independent 
engineers.  The  differences  between  the  two  estimates  revolved  around 
three  major  areas: 

•  The  first  included  a  discrepancy  of  almost  $270  million  in  the  cost  for 
ground  software  development.  The  program  office  estimated  the  cost 
at  $90  million  based  on  the  contractor’s  proposal  for  scaling  the 
software  and  productivity  rates  that  were  highly  optimistic.  AFCAA 
based  its  estimate  on  a  commercial  software  cost  model  using  DSMP 
and  SBIRS  High  historical  software  lines  of  code  growth  and  actual 
productivity  rates  from  the  Global  Positioning  System  program. 

•  The  second  difference  was  in  the  assembly  and  integration  and  testing 
estimates.  Compared  to  actual  integration  efforts  on  historical 
satellites  used  by  the  AFCAA,  the  program  office  estimate  to  integrate 
the  payloads  onto  the  satellite  bus  was  nearly  $132  million  less  than 
AFCAA’s  estimate. 


1  Aqua  collects  information  on  evaporation  from  the  oceans,  water  vapor  from  the 
atmosphere,  radioactive  energy  fluxes,  land  vegetation  cover,  and  land,  air,  and  water 
temperatures,  among  other  things.  Aura’s  mission  is  to  study  the  Earth’s  ozone,  air  quality, 
and  climate  focusing  exclusively  on  the  composition,  chemistry,  and  dynamics  of  the 
Earth’s  upper  and  lower  atmospheres.  The  Defense  Meteorological  Satellite  Program 
collects  weather  data  for  military  operations. 
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•  The  third  area  involved  the  systems  engineering  and  program 

management  costs  for  space  segment  development  and  production. 
AFCAA  used  actual  data  from  the  Aqua  and  Aura  satellites,  while  the 
program  office  relied  on  the  contractor’s  proposal — resulting  in  a 
difference  of  more  than  $130  million.  The  program  office’s  estimate 
was  lower  based  on  an  assumption  that  the  costs  for  systems 
engineering  and  program  management  would  be  reduced  by  almost 
50  percent  between  development  and  production.  AFCAA  stated 
concern  that  Aqua,  Aura,  and  DMSP  did  not  show  a  significant 
decrease  in  these  costs  over  time. 

Because  the  program  office’s  estimate  was  lower,  AFCAA  concluded  that 
the  program  office’s  cost  and  schedule  estimates  suffered  from  a  lack  of 
realism.  However,  the  results  of  AFCAA’ s  independent  cost  estimate  were 
not  used  by  the  program  office  officials  and  decision  makers. 

In  May  2004,  the  Under  Secretary  of  the  Air  Force  asked  the  CAIG  to 
prepare  an  independent  cost  estimate  for  the  NPOESS  program.  The 
estimate  was  completed  in  January  2005,  following  completion  of  the 
contractor’s  re-evaluation  of  the  program  baseline  in  November  2004.  The 
cost  estimate  focused  primarily  on  the  proposed  integration  schedule  of 
the  NPOESS  satellites.  This  estimate,  like  AFCAA’s  estimate  before  it,  was 
based  on  historical  cost  data  from  analogous  satellites  and  concluded  that 
the  program  office’s  proposed  integration  schedule  for  the  program  was 
unrealistic.  For  example,  the  program  office  proposed  an  integration 
schedule  for  the  first  NPOESS  satellite  that  was  about  half  the  time  needed 
for  an  analogous  satellite  that  had  almost  the  same  number  of  sensors.  In 
other  words,  the  NPOESS  program  estimated  that  it  would  integrate  close 
to  the  same  number  of  sensors  in  half  the  time.  Table  10  illustrates  how 
the  program  office  developed  its  integration  estimate  for  NPOESS,  which 
was  based  on  data  from  Aqua  satellites. 


Table  10:  Program  Office  Integration  Estimates  for  NPOESS 


Program 

Number  of 
sensors 

Months 
to  integrate 
based  on 
historical  data 

Months 
to  integrate 
sensor 

Deletion  of 
months  due  to 
unforeseen 
problems 

Months  to 
integrate 
without 
problems 

Months  to 
integrate 
sensor  without 
problems 

Aqua 

6 

31 

5.2 

-17 

14 

2.3 

NPOESS  (first 
satellite  integration  ) 

5 

26 

5.2 

N/A 

14 

2.8 

Source:  NPOESS  Executive  Committee  briefing,  January  2005. 
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The  program  office  relied  on  actual  data  for  Aqua,  with  no  unforeseen 
problems  as  the  basis  for  estimating  the  amount  of  time  needed  to 
integrate  NPOESS  sensors  on  the  first  satellite,  rather  than  using  historical 
data  that  would  have  yielded  an  estimate  of  26  months  to  integrate  five 
sensors.  The  program  office  and  the  contractor  contended  that  a  novel 
approach  was  being  taken  to  satellite  integration  on  the  NPOESS  program. 
The  CAIG  disagreed  with  this  contention,  stating  that  the  proposed 
integration  approach  was  not  really  novel  because  the  use  of  a  test  bed 
model  is  a  common  tool  used  by  satellite  programs  and  would  not  yield 
the  significant  savings  asserted  by  the  program  office.  The  CAIG,  instead, 
estimated  25  months  for  integrating  five  sensors  based  on  Aqua,  Aura,  and 
DMSP  historical  data.  As  a  result,  the  CAIG’s  estimate  was  almost  double 
the  program  office’s.  The  CAIG  also  expressed  concern  to  program 
officials  that  the  integration  schedule  was  severely  underestimated  and 
that  the  difference  between  the  program  office  estimate  and  the  CAIG’s 
estimate  was  more  than  6  years. 

The  program  office’s  2003  estimate  of  $7.2  billion  has  been  shown  to  be 
highly  unrealistic,  with  significant  cost  overruns  and  schedule  delays — 
thus  far — for  sensor  development  only.  Overall  satellite  integration  efforts 
have  been  delayed  due  to  the  problems  experienced  in  development  of  the 
sensors.  In  June  2006,  the  Office  of  the  Secretary  of  Defense  completed  the 
Nunn-McCurdy  process  and  certified  a  restructured  program  that  reduced 
the  number  of  satellites  to  be  developed — from  six  to  four,  with  the  first 
launch  being  delayed  to  2013  from  2009.  Cost  has  grown  from  the  original 
estimate  of  nearly  $7.2  billion  to  over  $11.4  billion — approximately  a 
60  percent  increase. 


SBIRS  High 


On  the  SBIRS  High  program,  the  program  office  and  AFCAA  predicted 
cost  growth  as  early  as  1996,  when  the  program  was  initiated.  While  both 
estimates  at  that  time  were  close  ($5.7  billion  in  2006  dollars  by  the 
program  office  and  $5.6  billion  in  2006  dollars  by  AFCAA),  both  were 
much  more  than  the  contractor’s  estimated  costs.  Nevertheless,  the 
program  was  subsequently  budgeted  at  $3.6  billion  by  the  program  office, 
almost  $2  billion  less  than  the  AFCAA  or  program  office  estimate.  The 
CAIG  stated  that  the  SBIRS  program  assumed  savings  under  TSPR  that 
simply  did  not  materialize.  SBIRS  program  officials  also  planned  on 
savings  from  simply  rehosting  existing  legacy  software,  but  those  savings 
were  not  realized  because  the  all  software  eventually  was  rewritten. 
Instead,  it  took  2  years  longer  than  planned  to  complete  the  first  increment 
of  software. 
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Savings  were  also  assumed  by  the  contractor  in  the  area  of  systems 
engineering.  The  SBIRS  High  contractor  initially  estimated  using  fewer 
systems  engineers,  even  though  historical  data  showed  programs  similar 
to  SBIRS  High  relied  on  three  to  almost  four  times  the  number  of  system 
engineers.  Some  of  the  tasks  dropped  from  the  systems  engineering  effort 
included  verification  and  cycling  of  requirements  because  the  government 
assumed  that  the  contractor  would  perform  these  activities  with  little  or 
no  oversight.  The  contractor  also  held  the  same  requirements  for  its 
subcontractors,  resulting  in  a  program  with  limited  systems  engineering. 
The  lack  of  systems  engineers  has  led  to  latent  design  flaws  and 
substantially  more  integration  and  testing  than  planned  because  no  one 
knew  what  had  gone  wrong  when  components  began  to  fail  during  testing. 
This  large  amount  of  rework  and  troubleshooting  has  led  to  substantial 
cost  and  schedule  increases. 

In  2005,  the  CAIG  reviewed  the  SBIRS  High  production  program  including 
estimating  the  cost  to  develop  geosynchronous  earth  orbiting  (GEO) 
satellites  3-5  as  clones  of  GEOs  1  and  2  in  order  to  determine  the  cost 
growth  incurred  by  the  production  program  since  2002.  The  CAIG’s 
analysis  projected  a  25  percent  Nunn-McCurdy  breach  in  average 
procurement  unit  cost  as  a  result  of  contractor  cost  and  schedule 
performance  being  markedly  worse  than  those  experienced  on  historical 
satellite  programs.  In  addition,  the  CAIG  found  that  government  actions  to 
date  have  been  ineffective  in  controlling  cost  and  schedule  growth.  The 
program  office,  on  the  other  hand,  showed  a  much  lower  cost  estimate  for 
the  production  cost  of  GEO  satellites  3-5,  as  seen  in  table  11. 


Table  11:  SBIRS  High  GEO  3-5  Procurement  Funding  Analysis 


Millions  of  then-year  dollars 

CAIG  Program 

Cost  baseline  estimate  office 


Delta  Delta  % 


Three  individual  $2,892  $2,027  $865  43% 

satellite 

procurements 


Source:  CAIG  and  GAO  analysis. 


The  CAIG  based  its  estimate  on  contractor  data  for  prime  contractor 
systems  engineering  and  program  management,  and  payload  integration 
assembly  and  test,  which  showed  substantial  increases  in  the  period  of 
performance,  staffing  levels,  and  hourly  rates  over  initial  estimates.  In 
addition,  the  CAIG’s  estimate  reflected  a  contractual  change  from  a  shared 
fee  pool  to  a  traditional  prime  contractor/subcontractor  relationship. 
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The  CAIG  expressed  concern  that  despite  restructuring  and  rebaselining 
the  program,  SBIRS  High  has  struggled  unabated  since  contract  award. 
The  CAIG  also  cautioned  that  rebaselining  would  only  allow  the  program 
to  hide  problems  in  the  short  term.  For  example,  the  CAIG  reported  that 
earned  value  management  data  showed  GEO  costs  were  following  the 
same  downward  trend  as  the  HEO  portion  of  the  program,  which  meant 
that  additional  cost  and  schedule  delays  were  possible. 
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OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
6000  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-6000 


NETWORKS  AND  INFORMATION 
INTEGRATION 

NOV  0  9  2006- 


ms.  Cristina  T.  Chaplain 

Director,  Acquisition  and  Sourcing  Management 

U.S.  Government  Accountability  Office 

441  G  Street,  NW 

Washington,  DC  20548 

Dear  Ms.  Chaplain: 

Thank  you  for  the  opportunity  to  comment  on  the  GAO  Draft  Report,  GAO-07-96 
entitled  “Space  Acquisitions:  DoD  Needs  to  Take  More  Action  to  Address  Unrealistic 
Initial  Cost  Estimates  of  Space  Systems,  dated  October  13, 2006  (GAO  Code  120554)”. 

I  concur  with  the  overall  findings  of  the  report,  and  have  enclosed  comments  to  your 
specific  recommendations. 

Again,  thank  you  for  this  opportunity  to  comment  on  your  report. 

Sincerely, 


^0$*d** - - 


John  R.  London 

Deputy  Assistant  Secretary  of  Defense 
_3ISR  &  IT  Acquisition) 


Enclosures: 
As  stated 
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GAO  DRAFT  REPORT  DATED  OCTOBER  13,  2006 
GAO-07-96  (GAO  CODE  120554) 

"SPACE  ACQUISITIONS:  DOD  NEEDS  TO  TAKE  MORE 
ACTION  TO  ADDRESS  UNREALISTIC  INITIAL  COST 
ESTIMATES  OF  SPACE  SYSTEMS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 

RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary  of  Defense 
direct  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  or  the 
Secretary  of  the  Air  Force,  as  appropriate,  to  increase  accountability  and  transparency 
of  decisions  in  space  programs  where  an  independent  estimate  produced  by  the  Cost 
Analysis  Improvement  Group  (CAIG)  or  Air  Force  Cost  Analysis  Agency  (AFCAA)  is 
not  chosen,  require  officials  involved  in  milestone  decisions  to  document  and  justify  the 
reasons  for  their  choice  and  the  differences  between  the  program  cost  estimate  and  the 
independent  cost  estimate,  (p.  21/GAO  Draft  Report) 

DOD  RESPONSE:  Partially  Concur.  At  both  the  development  and  the  production 
Milestones  for  all  Acquisition  Category  (ACAT)  I  programs  U.S.  Code  Title  10  - 
Armed  Forces  requires  the  Milestone  Decision  Authority  (MDA)  [USD(AT&L)j  to  be 
informed  by  an  independently  developed  life  cycle  cost  estimate  before  making  a 
decision  on  how  to  proceed.  The  OSD  CAIG  is  charged  with  developing  this  estimate 
and,  in  practice,  presents  their  findings  to  the  Defense  Acquisition  Board  (DAB)  when 
they  meet  to  advise  the  MDA.  Additionally,  the  OSD  CAIG  formally  documents  its 
Independent  Cost  Estimate  (ICE)  in  a  report  to  the  MDA.  The  complex  decision  to 
determine  the  cost  figure  used  as  a  basis  for  funding  and  to  evaluate  future  program 
performance  must  weigh  many  competing  factors  that  are  often  qualitative  in  nature. 

As  with  all  other  acquisition  related  decisions,  the  decision  is  the  MDA’s  alone,  and 
although  thoroughly  discussed  with  the  MDA  advisors  during  the  DAB  meeting  and 
clearly  documented  in  the  Acquisition  Decision  Memorandum  (ADM),  documenting 
the  explicit  justification  will  reduce  the  MDA’s  future  decision-making  flexibility. 
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RECOMMENDATION  2:  The  GAO  recommended  that  the  Secretary  of  Defense 
direct  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  or  the 
Secretary  of  the  Air  Force,  as  appropriate,  to  better  ensure  investment  decisions  for 
space  programs  are  knowledge-based,  instill  processes  and  tools  necessary  to  ensure 
lessons  learned  are  incorporated  into  future  estimates,  (p.  21/GAO  Draft  Report) 

POD  RESPONSE:  Concur  with  this  recommendation.  DoD  also  concurs  with  the 
following  recommendations  in  the  report: 

•  Conducting  post-mortem  reviews  of  past  space  program  cost  estimates  (program 
office  and  independent  cost  estimates)  to  measure  cost  estimating  effectiveness  and 
to  track  and  record  cost  estimating  mistakes. 

DoD  Response:  Concur.  The  OSD  CAIG  has  an  established  process  whereby  they 
meet  with  the  key  members  of  the  National  Security  space  cost  analysis  community  to 
discuss  and  evaluate  the  outcomes  following  ACAT  I  space  program  Milestone  or  Key 
Decision  Point  (UDP)  DAB-level  reviews.  The  purpose  of  this  meeting  is  to  provide 
visibility  to  the  other  members  of  the  National  Security  space  cost  analysis  community 
on  how  the  CAIG  approaches  ICE  development  and  to  give  the  community  an 
opportunity  to  provide  feedback  to  the  CAIG  on  how  to  improve  their  processes.  The 
OSD  CAIG  will  work  in  the  future  to  incorporate,  within  this  existing  framework,  peer 
reviews  of  the  associated  program  office  estimate.  The  OSD  CAIG,  as  required  by  the 
DoD’s  space  acquisition  regulations,  tracks  and  documents  their  ICEs  against  current 
program  office  estimates  as  each  ACAT  I  program  proceeds  through  its  development 
and  production  phases. 

•  Developing  a  centralized  cost  estimating  database  that  provides  realistic  and 
credible  data  to  cost  estimators. 

DoD  Response:  Concur.  Several  groups,  including  the  OSD  CAIG  sponsored 
National  Security  Space  Cost  Analysis  Symposium,  Consortium  on  Space  Technology 
Estimating  Research  (CoSTER),  and  the  National  Reconnaissance  Office  (NRO)  led 
cost  integrated  process  teams  (IPTs),  have  been  working  to  develop  a  database  of 
historical  space  program  costs  available  to  the  community  as  a  whole  for  model 
development  and  estimate  preparation.  Additionally,  the  OSD  CAIG  in  conjunction 
with  USD(AT&L)  and  approved  by  the  National  Security  space  cost  community,  has 
reestablished  a  common  space  program  work  breakdown  structure  (WBS),  incorporated 
in  the  latest  version  of  Military  Handbook  881,  that  supports  the  various  estimating 
methodologies  employed  by  the  space  cost  community.  This  recently  adopted  standard 
will  be  used  as  a  basis  for  all  future  space  program  cost  data  collection. 

•  Establishing  protocols  by  which  cost  estimators  working  with  the  NRO  can  share 
data  with  the  DOD  space  cost  estimating  community  while  still  maintaining 
appropriate  security  over  classified  data. 
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DoD  Response:  Concur.  Through  the  common  database  development  process 
described  above,  the  community,  within  security  constraints,  is  working  to  make 
historical  program  cost  data  as  widely  available  as  possible. 

•  Ensuring  estimates  are  updated  as  major  events  occur  within  a  program  that  could 
have  a  material  impact  on  cost,  such  as  budget  reductions,  integration  problems, 
hardware/software  quality  problems,  etc. 

DoD  Response:  Concur.  Updating  estimates,  including  independently  developed  cost 
estimates,  more  frequently  than  only  at  designated  KDPs  is  clearly  helpful  to  inform 
budgets  and  support  program  resource  adjustment  decisions.  However,  it  is  important 
not  to  mandate  updates  as,  by  their  nature,  they  should  be  program  and  program  phase 
dependent. 

RECOMMENDATION  3:  The  GAO  recommended  that  the  Secretary  of  Defense 
direct  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  or  the 
Secretary  of  the  Air  Force,  as  appropriate,  to  optimize  analysis  and  collaboration 
within  the  space  cost  estimating  community,  clearly  articulate  the  roles  and 
responsibilities  of  the  various  Air  Force  cost  estimating  organizations,  and  ensure  that 
space  system  cost  estimators  are  organized  so  that  the  Air  Force  can  gain  the  most 
from  their  knowledge  and  expertise.  In  taking  these  actions  for  programs  for  which  no 
independent  estimate  is  developed  by  the  DoD  CAIG,  consider  assigning  the  Air  Force 
Cost  Analysis  Agency  (AFCAA)  with  responsibility  for  the  development  of 
independent  cost  estimates  for  space  system  acquisitions,  since  is  outside  the 
acquisition  chain  of  command  and  therefore  more  likely  to  be  unbiased  and  not 
pressured  to  produce  optimistic  estimates,  (p.  21  GAO  Draft  Report) 

DOD  RESPONSE:  Concur.  The  Air  Force  is  currently  updating  their  Cost  and 
Economics  Policy  Directive  (AFPD  65-5)  and  associated  Instructions  to  further  clarify 
the  roles  and  responsibilities  of  their  space  cost  analysis  organizations.  The  purpose  of 
this  policy  also  addresses  Air  Force  goals  to  optimize  analysis  and  collaboration  thus 
making  the  best  use  of  the  limited  number  of  qualified  and  experienced  space  program 
cost  analysts. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 
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